[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:52:31 dist-test-slave-ktng krb5kdc[20207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:52:31 dist-test-slave-ktng krb5kdc[20207](info): set up 2 sockets
Sep 26 02:52:31 dist-test-slave-ktng krb5kdc[20207](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 02:52:32.704644 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.008s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:52:32 dist-test-slave-ktng krb5kdc[20207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:52:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:52:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:52:32.884475 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:33259
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:33259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:33.308804 20221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:33.309208 20221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:33.309423 20221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:33.318795 20221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:52:33.319005 20221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:33.319149 20221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:33.319367 20221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:52:33.319487 20221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:52:33.332268 20221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:33259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:33259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:33.333789 20221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:33.335596 20221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:33.359023 20227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:33.359371 20228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:33.359895 20230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:33.360869 20221 server_base.cc:1034] running on GCE node
I20260926 02:52:33.361850 20221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:33.363547 20221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:33.364804 20221 hybrid_clock.cc:648] HybridClock initialized: now 1790391153364720 us; error 30 us; skew 500 ppm
I20260926 02:52:33.365260 20221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:33 dist-test-slave-ktng krb5kdc[20207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391153, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:33.374166 20221 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:52:33.376266 20221 webserver.cc:460] Webserver started at http://127.19.185.254:34075/ using document root <none> and password file <none>
I20260926 02:52:33.377046 20221 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:33.377205 20221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:33.377516 20221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:33.381300 20221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "cf0bbd3996064c2bb140c0a486c169c9"
format_stamp: "Formatted at 2026-09-26 02:52:33 on dist-test-slave-ktng"
server_key: "6a8c2fd6d2ec0b34e688cdda12db7b04"
server_key_iv: "bb64444f82ca4f0fe0f1c4558de8adf3"
server_key_version: "encryptionkey@0"
I20260926 02:52:33.383522 20221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "cf0bbd3996064c2bb140c0a486c169c9"
format_stamp: "Formatted at 2026-09-26 02:52:33 on dist-test-slave-ktng"
server_key: "6a8c2fd6d2ec0b34e688cdda12db7b04"
server_key_iv: "bb64444f82ca4f0fe0f1c4558de8adf3"
server_key_version: "encryptionkey@0"
I20260926 02:52:33.390233 20221 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:52:33.394819 20237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:33.396481 20221 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:52:33.396749 20221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "cf0bbd3996064c2bb140c0a486c169c9"
format_stamp: "Formatted at 2026-09-26 02:52:33 on dist-test-slave-ktng"
server_key: "6a8c2fd6d2ec0b34e688cdda12db7b04"
server_key_iv: "bb64444f82ca4f0fe0f1c4558de8adf3"
server_key_version: "encryptionkey@0"
I20260926 02:52:33.397043 20221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:33.423583 20221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:33.429452 20221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:33.429927 20221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:33.457206 20221 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:33259
I20260926 02:52:33.457227 20289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:33259 every 8 connection(s)
I20260926 02:52:33.458632 20221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:52:33.462685 20290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:33.463618 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 20221
I20260926 02:52:33.463953 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:52:33.466466 20199 external_mini_cluster.cc:1428] Setting key 40a605fcf8c6211ecca2e7f038f1512e
I20260926 02:52:33.482575 20290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9: Bootstrap starting.
I20260926 02:52:33.485703 20290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:33.486730 20290 log.cc:824] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9: Log is configured to *not* fsync() on all Append() calls
I20260926 02:52:33.489804 20290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9: No bootstrap required, opened a new log
Sep 26 02:52:33 dist-test-slave-ktng krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:33.500586 20290 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0bbd3996064c2bb140c0a486c169c9" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 33259 } }
I20260926 02:52:33.501509 20290 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:33.501713 20290 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0bbd3996064c2bb140c0a486c169c9, State: Initialized, Role: FOLLOWER
I20260926 02:52:33.502624 20290 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0bbd3996064c2bb140c0a486c169c9" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 33259 } }
I20260926 02:52:33.502967 20290 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:52:33.503252 20290 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:52:33.503574 20290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:33.506830 20290 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0bbd3996064c2bb140c0a486c169c9" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 33259 } }
I20260926 02:52:33.507532 20290 leader_election.cc:302] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf0bbd3996064c2bb140c0a486c169c9; no voters:
I20260926 02:52:33.508803 20290 leader_election.cc:288] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:52:33.509100 20295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:33.512460 20290 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:52:33.512693 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:33.470971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.19.185.254:33259)
0926 02:52:33.471824 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:33.471842 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:33.472773 (+ 931us) server_negotiation.cc:407] Connection header received
0926 02:52:33.481276 (+ 8503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:33.481300 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:33.481683 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:33.482080 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:33.485112 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:33.488452 (+ 3340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:33.491743 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:33.492454 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:33.492679 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:33.500714 (+ 8035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:33.500727 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:33.500746 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:33.500817 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:33.505882 (+ 5065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:33.506868 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:33.506877 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:33.506885 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:33.507250 (+ 365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:33.507995 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:33.508002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:33.508008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:33.508683 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:33.509171 (+ 488us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:33.510463 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0926 02:52:33.510943 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":362,"threads_started":1}
I20260926 02:52:33.515735 20295 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [term 1 LEADER]: Becoming Leader. State: Replica: cf0bbd3996064c2bb140c0a486c169c9, State: Running, Role: LEADER
I20260926 02:52:33.516645 20295 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0bbd3996064c2bb140c0a486c169c9" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 33259 } }
I20260926 02:52:33.522610 20298 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf0bbd3996064c2bb140c0a486c169c9. Latest consensus state: current_term: 1 leader_uuid: "cf0bbd3996064c2bb140c0a486c169c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0bbd3996064c2bb140c0a486c169c9" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 33259 } } }
I20260926 02:52:33.523553 20298 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [sys.catalog]: This master's current role is: LEADER
I20260926 02:52:33.522292 20297 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf0bbd3996064c2bb140c0a486c169c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0bbd3996064c2bb140c0a486c169c9" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 33259 } } }
I20260926 02:52:33.530165 20297 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9 [sys.catalog]: This master's current role is: LEADER
I20260926 02:52:33.532797 20307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:52:33.542303 20307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:52:33.557528 20307 catalog_manager.cc:1347] Generated new cluster ID: 618754c3b8334c9fbc43becdf8be631a
I20260926 02:52:33.557824 20307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:52:33.572492 20307 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:52:33.574150 20307 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:52:33.593914 20307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf0bbd3996064c2bb140c0a486c169c9: Generated new TSK 0
I20260926 02:52:33.595121 20307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:52:33.640321 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:33259
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:33.992937 20318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:33.993225 20318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:33.993393 20318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:34.000471 20318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:34.000568 20318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:34.000790 20318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:52:34.010483 20318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:33259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:34.011873 20318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:34.013298 20318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:34.026239 20323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:34.026587 20326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:34.026669 20324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:34.027827 20318 server_base.cc:1034] running on GCE node
I20260926 02:52:34.028422 20318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:34.029345 20318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:34.030580 20318 hybrid_clock.cc:648] HybridClock initialized: now 1790391154030508 us; error 63 us; skew 500 ppm
I20260926 02:52:34.030906 20318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:34 dist-test-slave-ktng krb5kdc[20207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391154, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:34.034168 20318 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:52:34.035593 20318 webserver.cc:460] Webserver started at http://127.19.185.193:41061/ using document root <none> and password file <none>
I20260926 02:52:34.036279 20318 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:34.036425 20318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:34.036736 20318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:34.039769 20318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "bff98a184585420080714823ae09c6e2"
format_stamp: "Formatted at 2026-09-26 02:52:34 on dist-test-slave-ktng"
server_key: "215afa0c29ef7286e2847e33af8ce329"
server_key_iv: "67279ee4b123d2b443b5b4290955199c"
server_key_version: "encryptionkey@0"
I20260926 02:52:34.040580 20318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "bff98a184585420080714823ae09c6e2"
format_stamp: "Formatted at 2026-09-26 02:52:34 on dist-test-slave-ktng"
server_key: "215afa0c29ef7286e2847e33af8ce329"
server_key_iv: "67279ee4b123d2b443b5b4290955199c"
server_key_version: "encryptionkey@0"
I20260926 02:52:34.045722 20318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 02:52:34.049115 20334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:34.050508 20318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:52:34.050698 20318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "bff98a184585420080714823ae09c6e2"
format_stamp: "Formatted at 2026-09-26 02:52:34 on dist-test-slave-ktng"
server_key: "215afa0c29ef7286e2847e33af8ce329"
server_key_iv: "67279ee4b123d2b443b5b4290955199c"
server_key_version: "encryptionkey@0"
I20260926 02:52:34.050901 20318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:34.073356 20318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:34.077718 20318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:34.078028 20318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:34.078986 20318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:52:34.080454 20318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:34.080564 20318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:34.080716 20318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:34.080788 20318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:34.119212 20318 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:41309
I20260926 02:52:34.119256 20447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:41309 every 8 connection(s)
I20260926 02:52:34.120517 20318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:52:34.127514 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 20318
I20260926 02:52:34.127746 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:52:34.128392 20199 external_mini_cluster.cc:1428] Setting key 0b70d02603c558acc8ae541985a6c903
Sep 26 02:52:34 dist-test-slave-ktng krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391154, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:34.140249 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:34.123067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:42907 (local address 127.19.185.254:33259)
0926 02:52:34.123557 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:34.123561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:34.124557 (+ 996us) server_negotiation.cc:407] Connection header received
0926 02:52:34.126397 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:34.126402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:34.126493 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:34.126615 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:34.128249 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:34.129102 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:34.130726 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:34.131126 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:34.131263 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:34.134608 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:34.134615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:34.134618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:34.134649 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:34.137197 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:34.137901 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:34.137907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:34.137912 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:34.138046 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:34.138486 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:34.138495 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:34.138497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:34.138745 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:34.138905 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:34.139643 (+ 738us) server_negotiation.cc:299] Negotiation successful
0926 02:52:34.139872 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":265,"threads_started":1}
I20260926 02:52:34.141242 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:34.123707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:33259 (local address 127.19.185.193:42907)
0926 02:52:34.124383 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:34.124426 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:34.126137 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:34.126862 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:34.126869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:34.127433 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:34.128084 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:34.128102 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:34.129278 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:34.129282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:34.130569 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:34.130580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:34.131378 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:34.131387 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:34.131627 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:34.132296 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:34.132324 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:34.134392 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:34.137336 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:34.137346 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:34.137361 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:34.137774 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:34.138173 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:34.138177 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:34.138181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:34.138365 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:34.138883 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:34.138890 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:34.139220 (+ 330us) client_negotiation.cc:769] Sending connection context
0926 02:52:34.139541 (+ 321us) client_negotiation.cc:240] Negotiation successful
0926 02:52:34.140097 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":254,"threads_started":1}
I20260926 02:52:34.142889 20448 heartbeater.cc:347] Connected to a master server at 127.19.185.254:33259
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
I20260926 02:52:34.143214 20448 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:34.144004 20448 heartbeater.cc:511] Master 127.19.185.254:33259 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
I20260926 02:52:34.146451 20255 ts_manager.cc:194] Registered new tserver with Master: bff98a184585420080714823ae09c6e2 (127.19.185.193:41309)
I20260926 02:52:34.147984 20255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:42907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:52:34.166937 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:33259
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:34.564201 20454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:34.564473 20454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:34.564634 20454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:34.571870 20454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:34.572013 20454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:34.572206 20454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:52:34.581844 20454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:33259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:34.583132 20454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:34.584486 20454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:34.597816 20459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:34.597867 20460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:34.598021 20462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:34.598579 20454 server_base.cc:1034] running on GCE node
I20260926 02:52:34.599187 20454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:34.600178 20454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:34.601454 20454 hybrid_clock.cc:648] HybridClock initialized: now 1790391154601387 us; error 41 us; skew 500 ppm
I20260926 02:52:34.601795 20454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:34 dist-test-slave-ktng krb5kdc[20207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391154, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:34.605041 20454 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:52:34.606587 20454 webserver.cc:460] Webserver started at http://127.19.185.194:37929/ using document root <none> and password file <none>
I20260926 02:52:34.607343 20454 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:34.607501 20454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:34.607847 20454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:34.610783 20454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "35e98834a6f045db83c0afd6dd1ce8c5"
format_stamp: "Formatted at 2026-09-26 02:52:34 on dist-test-slave-ktng"
server_key: "21845bf12345280199ad5aaac48618b8"
server_key_iv: "17bdcc7a9331ac396692987231392034"
server_key_version: "encryptionkey@0"
I20260926 02:52:34.611636 20454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "35e98834a6f045db83c0afd6dd1ce8c5"
format_stamp: "Formatted at 2026-09-26 02:52:34 on dist-test-slave-ktng"
server_key: "21845bf12345280199ad5aaac48618b8"
server_key_iv: "17bdcc7a9331ac396692987231392034"
server_key_version: "encryptionkey@0"
I20260926 02:52:34.617053 20454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:52:34.620503 20469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:34.622051 20454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 02:52:34.622224 20454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "35e98834a6f045db83c0afd6dd1ce8c5"
format_stamp: "Formatted at 2026-09-26 02:52:34 on dist-test-slave-ktng"
server_key: "21845bf12345280199ad5aaac48618b8"
server_key_iv: "17bdcc7a9331ac396692987231392034"
server_key_version: "encryptionkey@0"
I20260926 02:52:34.622412 20454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:34.650718 20454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:34.654873 20454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:34.655184 20454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:34.656188 20454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:52:34.657622 20454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:34.657732 20454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:34.657883 20454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:34.657975 20454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:34.692122 20454 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:42595
I20260926 02:52:34.692167 20581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:42595 every 8 connection(s)
I20260926 02:52:34.693451 20454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:52:34.694120 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 20454
I20260926 02:52:34.694332 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:52:34.694916 20199 external_mini_cluster.cc:1428] Setting key 0bae71db096f022bb3877080eeac3292
Sep 26 02:52:34 dist-test-slave-ktng krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391154, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
I20260926 02:52:34.716265 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:34.696408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:42077 (local address 127.19.185.254:33259)
0926 02:52:34.696844 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:34.696851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:34.698004 (+ 1153us) server_negotiation.cc:407] Connection header received
0926 02:52:34.699911 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:34.699916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:34.699998 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:34.700131 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:34.701708 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:34.702492 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:34.705133 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:34.705559 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:34.705662 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:34.710129 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:34.710135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:34.710138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:34.710171 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:34.713003 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:34.713845 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:34.713849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:34.713851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:34.713980 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:34.714507 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:34.714510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:34.714512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:34.714794 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:34.714944 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:34.715765 (+ 821us) server_negotiation.cc:299] Negotiation successful
0926 02:52:34.716000 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":228,"threads_started":1}
I20260926 02:52:34.717829 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:34.697116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:33259 (local address 127.19.185.194:42077)
0926 02:52:34.697849 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:34.697886 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:34.699655 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:34.700366 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:34.700374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:34.700945 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:34.701518 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:34.701535 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:34.702646 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:34.702654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:34.704963 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:34.704985 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:34.705794 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:34.705811 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:34.706147 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:34.706876 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:34.706911 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:34.709796 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:34.713148 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:34.713158 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:34.713175 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:34.713686 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:34.714095 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:34.714100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:34.714104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:34.714337 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:34.714922 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:34.714930 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:34.715350 (+ 420us) client_negotiation.cc:769] Sending connection context
0926 02:52:34.715666 (+ 316us) client_negotiation.cc:240] Negotiation successful
0926 02:52:34.716246 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":320,"threads_started":1}
I20260926 02:52:34.719411 20582 heartbeater.cc:347] Connected to a master server at 127.19.185.254:33259
I20260926 02:52:34.719740 20582 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:34.720664 20582 heartbeater.cc:511] Master 127.19.185.254:33259 requested a full tablet report, sending...
I20260926 02:52:34.722348 20255 ts_manager.cc:194] Registered new tserver with Master: 35e98834a6f045db83c0afd6dd1ce8c5 (127.19.185.194:42595)
I20260926 02:52:34.723271 20255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:42077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:52:34.740859 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:33259
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:35.107959 20588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:35.108349 20588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:35.108549 20588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:35.116024 20588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:35.116166 20588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:35.116335 20588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:52:35.129439 20588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:33259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:35.130739 20588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:35.132118 20588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:35.146245 20596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:35.146200 20594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:35.146286 20593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:35.151103 20448 heartbeater.cc:503] Master 127.19.185.254:33259 was elected leader, sending a full tablet report...
I20260926 02:52:35.152354 20588 server_base.cc:1034] running on GCE node
I20260926 02:52:35.153154 20588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:35.154269 20588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:35.155534 20588 hybrid_clock.cc:648] HybridClock initialized: now 1790391155155475 us; error 41 us; skew 500 ppm
I20260926 02:52:35.155898 20588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:35 dist-test-slave-ktng krb5kdc[20207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391155, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:35.159070 20588 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:52:35.160653 20588 webserver.cc:460] Webserver started at http://127.19.185.195:35231/ using document root <none> and password file <none>
I20260926 02:52:35.161734 20588 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:35.161885 20588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:35.162304 20588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:35.166733 20588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "1c75a2e0b5774fcfa1b08fce42863ef1"
format_stamp: "Formatted at 2026-09-26 02:52:35 on dist-test-slave-ktng"
server_key: "6bfe465311aa00c623fe64baacf48a0a"
server_key_iv: "2da6fb1d8620439c7daa28ebacb21298"
server_key_version: "encryptionkey@0"
I20260926 02:52:35.167645 20588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "1c75a2e0b5774fcfa1b08fce42863ef1"
format_stamp: "Formatted at 2026-09-26 02:52:35 on dist-test-slave-ktng"
server_key: "6bfe465311aa00c623fe64baacf48a0a"
server_key_iv: "2da6fb1d8620439c7daa28ebacb21298"
server_key_version: "encryptionkey@0"
I20260926 02:52:35.172868 20588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 02:52:35.176267 20603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:35.177554 20588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:52:35.177744 20588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "1c75a2e0b5774fcfa1b08fce42863ef1"
format_stamp: "Formatted at 2026-09-26 02:52:35 on dist-test-slave-ktng"
server_key: "6bfe465311aa00c623fe64baacf48a0a"
server_key_iv: "2da6fb1d8620439c7daa28ebacb21298"
server_key_version: "encryptionkey@0"
I20260926 02:52:35.177954 20588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:35.198410 20588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:35.202526 20588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:35.202814 20588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:35.203799 20588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:52:35.205096 20588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:35.205199 20588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:35.205344 20588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:35.205415 20588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:35.241255 20588 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:45821
I20260926 02:52:35.241302 20715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:45821 every 8 connection(s)
I20260926 02:52:35.242660 20588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:52:35.247882 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 20588
I20260926 02:52:35.248075 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:52:35.248507 20199 external_mini_cluster.cc:1428] Setting key 41d46c793b802aec09d44e9086dea020
Sep 26 02:52:35 dist-test-slave-ktng krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391155, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:35.263491 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:35.245293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:33557 (local address 127.19.185.254:33259)
0926 02:52:35.245775 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:35.245782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:35.246790 (+ 1008us) server_negotiation.cc:407] Connection header received
0926 02:52:35.248777 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:35.248783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:35.248883 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:35.249017 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:35.250925 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:35.251967 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:35.253766 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:35.254174 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:35.254274 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:35.257871 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:35.257875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:35.257879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:35.257913 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:35.260683 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:35.261385 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:35.261389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:35.261391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:35.261513 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:35.261870 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:35.261873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:35.261875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:35.262124 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:35.262230 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:35.262842 (+ 612us) server_negotiation.cc:299] Negotiation successful
0926 02:52:35.263074 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":215,"threads_started":1}
I20260926 02:52:35.264585 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:35.245931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:33259 (local address 127.19.185.195:33557)
0926 02:52:35.246620 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:35.246668 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:35.248473 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:35.249249 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:35.249259 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:35.250022 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:35.250743 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:35.250762 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:35.252158 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:35.252165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:35.253616 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:35.253628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:35.254364 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:35.254369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:35.254643 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:35.255382 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:35.255409 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:35.257662 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:35.260809 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:35.260816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:35.260829 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:35.261280 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:35.261607 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:35.261610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:35.261612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:35.261780 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:35.262224 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:35.262229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:35.262533 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 02:52:35.262769 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 02:52:35.263180 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":265,"threads_started":1}
I20260926 02:52:35.265957 20716 heartbeater.cc:347] Connected to a master server at 127.19.185.254:33259
I20260926 02:52:35.266225 20716 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:35.266883 20716 heartbeater.cc:511] Master 127.19.185.254:33259 requested a full tablet report, sending...
I20260926 02:52:35.268389 20255 ts_manager.cc:194] Registered new tserver with Master: 1c75a2e0b5774fcfa1b08fce42863ef1 (127.19.185.195:45821)
I20260926 02:52:35.269217 20255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:33557
I20260926 02:52:35.277546 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:52:35 dist-test-slave-ktng krb5kdc[20207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:52:35 dist-test-slave-ktng krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:52:35.320986 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:35.305834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46222 (local address 127.19.185.193:41309)
0926 02:52:35.306510 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:35.306518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:35.306546 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:52:35.306661 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:35.306668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:35.306887 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:35.307024 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:35.308493 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:35.309472 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:35.310695 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:35.311246 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:35.311425 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:35.315447 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:35.315458 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:35.315472 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:35.315508 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:35.318426 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:35.319076 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:35.319083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:35.319086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:35.319209 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:35.319621 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:35.319625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:35.319627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:35.319947 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:35.320221 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:35.320407 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 02:52:35.320585 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":388,"threads_started":1}
W20260926 02:52:35.322305 20363 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46222
Sep 26 02:52:35 dist-test-slave-ktng krb5kdc[20207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:52:35 dist-test-slave-ktng krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:52:35.366456 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:35.352404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.19.185.193:41309)
0926 02:52:35.352638 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:35.352645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:35.352829 (+ 184us) server_negotiation.cc:407] Connection header received
0926 02:52:35.353043 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:35.353048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:35.353157 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:35.353268 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:35.354680 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:35.355565 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:35.356776 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:35.357319 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:35.357459 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:35.361533 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:35.361538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:35.361541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:35.361573 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:35.363966 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:35.364612 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:35.364624 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:35.364627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:35.364738 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:35.365092 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:35.365099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:35.365102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:35.365381 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:35.365619 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:35.365920 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 02:52:35.366144 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:52:35.368027 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 20318
I20260926 02:52:35.380913 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:35.368961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.19.185.193:41309)
0926 02:52:35.369139 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:35.369147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:35.369406 (+ 259us) server_negotiation.cc:407] Connection header received
0926 02:52:35.369577 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:35.369582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:35.369671 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:35.369769 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:35.370944 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:35.371695 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:35.373101 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:35.373503 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:35.373692 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:35.375951 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:35.375957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:35.375961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:35.375997 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:35.378255 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:35.379020 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:35.379027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:35.379030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:35.379163 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:35.379542 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:35.379548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:35.379551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:35.379804 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:35.380043 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:35.380343 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 02:52:35.380541 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 02:52:35.381611 20443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:52:35.466923 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 20318
I20260926 02:52:35.483778 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 20454
Sep 26 02:52:35 dist-test-slave-ktng krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:52:35.499804 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:35.484679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45160 (local address 127.19.185.194:42595)
0926 02:52:35.485246 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:35.485253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:35.485279 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:52:35.485337 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:35.485345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:35.485569 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:35.485749 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:35.486918 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:35.488004 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:35.489237 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:35.489800 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:35.489954 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:35.493918 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:35.493960 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:35.493974 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:35.494012 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:35.496793 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:35.497458 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:35.497465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:35.497468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:35.497593 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:35.497986 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:35.497993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:35.497996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:35.498351 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:35.498792 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:35.499127 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 02:52:35.499439 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":320,"threads_started":1}
I20260926 02:52:35.501081 20577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:52:35.583165 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 20454
I20260926 02:52:35.605509 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 20588
Sep 26 02:52:35 dist-test-slave-ktng krb5kdc[20207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:52:35.621138 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:35.606400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.19.185.195:45821)
0926 02:52:35.607076 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:35.607084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:35.607114 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:52:35.607178 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:35.607185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:35.607471 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:35.607607 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:35.608859 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:35.609851 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:35.611083 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:35.611719 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:35.611816 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:35.615977 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:35.615987 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:35.616001 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:35.616030 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:35.618790 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:35.619422 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:35.619428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:35.619431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:35.619544 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:35.619853 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:35.619855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:35.619857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:35.620167 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:35.620387 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:35.620664 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 02:52:35.620854 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":326,"threads_started":1}
I20260926 02:52:35.622510 20711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:52:35.705107 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 20588
I20260926 02:52:35.723285 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 20221
I20260926 02:52:35.724581 20285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:52:35.827936 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 20221
2026-09-26T02:52:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4247 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:52:35 dist-test-slave-ktng krb5kdc[20738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:52:35 dist-test-slave-ktng krb5kdc[20738](info): set up 2 sockets
Sep 26 02:52:35 dist-test-slave-ktng krb5kdc[20738](info): commencing operation
krb5kdc: starting...
W20260926 02:52:36.924105 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:52:37 dist-test-slave-ktng krb5kdc[20738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:52:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:52:37.102977 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:46757
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:46757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:37.419166 20752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:37.419517 20752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:37.419682 20752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:37.426743 20752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:52:37.426899 20752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:37.427099 20752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:37.427204 20752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:52:37.427273 20752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:52:37.437145 20752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:46757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:46757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:37.438578 20752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:37.439944 20752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:37.452630 20760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:37.452749 20758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:37.453394 20757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:37.453926 20752 server_base.cc:1034] running on GCE node
I20260926 02:52:37.454794 20752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:37.456334 20752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:37.457559 20752 hybrid_clock.cc:648] HybridClock initialized: now 1790391157457491 us; error 55 us; skew 500 ppm
I20260926 02:52:37.457940 20752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:37 dist-test-slave-ktng krb5kdc[20738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391157, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:37.461158 20752 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:52:37.462711 20752 webserver.cc:460] Webserver started at http://127.19.185.254:41789/ using document root <none> and password file <none>
I20260926 02:52:37.463456 20752 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:37.463599 20752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:37.463917 20752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:37.466835 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "3da0c21f770941aaa8acb3b23d68b1e8"
format_stamp: "Formatted at 2026-09-26 02:52:37 on dist-test-slave-ktng"
server_key: "61a2cb95e7b3662f20a47ba4c778fe79"
server_key_iv: "4844eccf205df42df5da6e845ef27d50"
server_key_version: "encryptionkey@0"
I20260926 02:52:37.467624 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "3da0c21f770941aaa8acb3b23d68b1e8"
format_stamp: "Formatted at 2026-09-26 02:52:37 on dist-test-slave-ktng"
server_key: "61a2cb95e7b3662f20a47ba4c778fe79"
server_key_iv: "4844eccf205df42df5da6e845ef27d50"
server_key_version: "encryptionkey@0"
I20260926 02:52:37.472759 20752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:52:37.476292 20767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:37.477645 20752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:52:37.477824 20752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "3da0c21f770941aaa8acb3b23d68b1e8"
format_stamp: "Formatted at 2026-09-26 02:52:37 on dist-test-slave-ktng"
server_key: "61a2cb95e7b3662f20a47ba4c778fe79"
server_key_iv: "4844eccf205df42df5da6e845ef27d50"
server_key_version: "encryptionkey@0"
I20260926 02:52:37.478030 20752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:37.492233 20752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:37.496447 20752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:37.496749 20752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:37.515430 20752 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:46757
I20260926 02:52:37.515419 20819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:46757 every 8 connection(s)
I20260926 02:52:37.516804 20752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:52:37.518929 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 20752
I20260926 02:52:37.519177 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:52:37.519824 20199 external_mini_cluster.cc:1428] Setting key 4b88e1bfcd994c050a8e518eed52d453
I20260926 02:52:37.520848 20820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:37.532847 20820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8: Bootstrap starting.
Sep 26 02:52:37 dist-test-slave-ktng krb5kdc[20738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:37.536859 20820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:37.538275 20820 log.cc:824] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8: Log is configured to *not* fsync() on all Append() calls
I20260926 02:52:37.541942 20820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8: No bootstrap required, opened a new log
I20260926 02:52:37.543129 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:37.521727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.19.185.254:46757)
0926 02:52:37.522498 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:37.522509 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:37.522546 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:52:37.524202 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:37.524224 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:37.524598 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:37.524997 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:37.526439 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:37.527807 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:37.529023 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:37.529710 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:37.529922 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:37.534043 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:37.534055 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:37.534074 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:37.534131 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:37.538216 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:37.539203 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:37.539209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:37.539215 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:37.539463 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:37.539901 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:37.539910 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:37.539916 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:37.540616 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:37.540905 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:37.541381 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 02:52:37.541796 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":371,"threads_started":1}
I20260926 02:52:37.550431 20820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da0c21f770941aaa8acb3b23d68b1e8" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46757 } }
I20260926 02:52:37.550781 20820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:37.550853 20820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da0c21f770941aaa8acb3b23d68b1e8, State: Initialized, Role: FOLLOWER
I20260926 02:52:37.551473 20820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da0c21f770941aaa8acb3b23d68b1e8" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46757 } }
I20260926 02:52:37.551679 20820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:52:37.551774 20820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:52:37.551892 20820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:37.553994 20820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da0c21f770941aaa8acb3b23d68b1e8" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46757 } }
I20260926 02:52:37.554582 20820 leader_election.cc:302] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3da0c21f770941aaa8acb3b23d68b1e8; no voters:
I20260926 02:52:37.555202 20820 leader_election.cc:288] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:52:37.555380 20825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:37.556165 20825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [term 1 LEADER]: Becoming Leader. State: Replica: 3da0c21f770941aaa8acb3b23d68b1e8, State: Running, Role: LEADER
I20260926 02:52:37.556638 20825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da0c21f770941aaa8acb3b23d68b1e8" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46757 } }
I20260926 02:52:37.557296 20820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:52:37.560204 20826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3da0c21f770941aaa8acb3b23d68b1e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da0c21f770941aaa8acb3b23d68b1e8" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46757 } } }
I20260926 02:52:37.561058 20826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [sys.catalog]: This master's current role is: LEADER
I20260926 02:52:37.560712 20827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3da0c21f770941aaa8acb3b23d68b1e8. Latest consensus state: current_term: 1 leader_uuid: "3da0c21f770941aaa8acb3b23d68b1e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da0c21f770941aaa8acb3b23d68b1e8" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46757 } } }
I20260926 02:52:37.561573 20827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8 [sys.catalog]: This master's current role is: LEADER
I20260926 02:52:37.563046 20834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:52:37.568475 20834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:52:37.577668 20834 catalog_manager.cc:1347] Generated new cluster ID: 3a3a034f1a7840f8a2fdd45c2c7fae5c
I20260926 02:52:37.577782 20834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:52:37.606297 20834 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:52:37.607745 20834 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:52:37.617430 20834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3da0c21f770941aaa8acb3b23d68b1e8: Generated new TSK 0
I20260926 02:52:37.618441 20834 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:52:37.675426 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46757
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:38.007892 20847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:38.008328 20847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:38.008512 20847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:38.020905 20847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:38.021080 20847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:38.021335 20847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:52:38.037195 20847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46757
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:38.038951 20847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:38.040606 20847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:38.054656 20852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:38.054692 20853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:38.054575 20855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:38.055835 20847 server_base.cc:1034] running on GCE node
I20260926 02:52:38.056496 20847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:38.057550 20847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:38.058833 20847 hybrid_clock.cc:648] HybridClock initialized: now 1790391158058778 us; error 32 us; skew 500 ppm
I20260926 02:52:38.059161 20847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:38 dist-test-slave-ktng krb5kdc[20738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391158, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:38.062564 20847 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:52:38.064179 20847 webserver.cc:460] Webserver started at http://127.19.185.193:44937/ using document root <none> and password file <none>
I20260926 02:52:38.064894 20847 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:38.064993 20847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:38.065253 20847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:38.068157 20847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "ce3dfcf5969e4396b009841436d66f22"
format_stamp: "Formatted at 2026-09-26 02:52:38 on dist-test-slave-ktng"
server_key: "9f1e36ccaa75ecf5f2c20ac8cd7263d0"
server_key_iv: "2ee1a11dd9dd3bbc8c7259f63ff5eedf"
server_key_version: "encryptionkey@0"
I20260926 02:52:38.068886 20847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "ce3dfcf5969e4396b009841436d66f22"
format_stamp: "Formatted at 2026-09-26 02:52:38 on dist-test-slave-ktng"
server_key: "9f1e36ccaa75ecf5f2c20ac8cd7263d0"
server_key_iv: "2ee1a11dd9dd3bbc8c7259f63ff5eedf"
server_key_version: "encryptionkey@0"
I20260926 02:52:38.074148 20847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:52:38.077606 20862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:38.079139 20847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 02:52:38.079360 20847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "ce3dfcf5969e4396b009841436d66f22"
format_stamp: "Formatted at 2026-09-26 02:52:38 on dist-test-slave-ktng"
server_key: "9f1e36ccaa75ecf5f2c20ac8cd7263d0"
server_key_iv: "2ee1a11dd9dd3bbc8c7259f63ff5eedf"
server_key_version: "encryptionkey@0"
I20260926 02:52:38.079572 20847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:38.095253 20847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:38.099445 20847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:38.099728 20847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:38.100667 20847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:52:38.101953 20847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:38.102025 20847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:38.102145 20847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:38.102207 20847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:38.134819 20847 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:36311
I20260926 02:52:38.134857 20974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:36311 every 8 connection(s)
I20260926 02:52:38.136127 20847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:52:38.144917 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 20847
I20260926 02:52:38.145172 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:52:38.145785 20199 external_mini_cluster.cc:1428] Setting key b5341ce6805fc6dfd8e820e2e75849fa
Sep 26 02:52:38 dist-test-slave-ktng krb5kdc[20738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391158, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:38.157112 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:38.138672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:37995 (local address 127.19.185.254:46757)
0926 02:52:38.139141 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:38.139145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:38.140088 (+ 943us) server_negotiation.cc:407] Connection header received
0926 02:52:38.141903 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:38.141908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:38.142013 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:38.142152 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:38.143708 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:38.144434 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:38.146159 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:38.146724 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:38.146875 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:38.151205 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:38.151210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:38.151213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:38.151246 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:38.153912 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:38.154667 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:38.154672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:38.154676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:38.154797 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:38.155238 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:38.155243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:38.155246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:38.155650 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:38.155814 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:38.156536 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 02:52:38.156751 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":242,"threads_started":1}
I20260926 02:52:38.158281 20978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:38.139124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:46757 (local address 127.19.185.193:37995)
0926 02:52:38.139938 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:38.139980 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:38.141631 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:38.142381 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:38.142390 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:38.142901 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:38.143532 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:38.143550 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:38.144621 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:38.144625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:38.146015 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:38.146027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:38.146952 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:38.146957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:38.147178 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:38.148125 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:38.148155 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:38.150958 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:38.154080 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:38.154090 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:38.154116 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:38.154537 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:38.154910 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:38.154916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:38.154919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:38.155118 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:38.155792 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:38.155801 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:38.156158 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 02:52:38.156419 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 02:52:38.156978 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":319,"threads_started":1}
I20260926 02:52:38.159974 20975 heartbeater.cc:347] Connected to a master server at 127.19.185.254:46757
I20260926 02:52:38.160328 20975 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:38.161250 20975 heartbeater.cc:511] Master 127.19.185.254:46757 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
I20260926 02:52:38.163844 20784 ts_manager.cc:194] Registered new tserver with Master: ce3dfcf5969e4396b009841436d66f22 (127.19.185.193:36311)
I20260926 02:52:38.165432 20784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:37995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:52:38.187676 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46757
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:38.501890 20981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:38.502172 20981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:38.502269 20981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:38.509405 20981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:38.509524 20981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:38.509640 20981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:52:38.519428 20981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46757
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:38.520709 20981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:38.521978 20981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:38.535689 20989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:38.535774 20986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:38.535774 20987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:38.536482 20981 server_base.cc:1034] running on GCE node
I20260926 02:52:38.537053 20981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:38.537945 20981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:38.539163 20981 hybrid_clock.cc:648] HybridClock initialized: now 1790391158539098 us; error 54 us; skew 500 ppm
I20260926 02:52:38.539572 20981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:38 dist-test-slave-ktng krb5kdc[20738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391158, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:38.546452 20981 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:52:38.547984 20981 webserver.cc:460] Webserver started at http://127.19.185.194:38569/ using document root <none> and password file <none>
I20260926 02:52:38.548681 20981 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:38.548838 20981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:38.549175 20981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:38.552205 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "8bb76518363e4fef8a35b679a2c498a9"
format_stamp: "Formatted at 2026-09-26 02:52:38 on dist-test-slave-ktng"
server_key: "5370188f04d6445e7f8ab3fd403fcd02"
server_key_iv: "c14d911197d6148821b5f590abf87dbc"
server_key_version: "encryptionkey@0"
I20260926 02:52:38.552997 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "8bb76518363e4fef8a35b679a2c498a9"
format_stamp: "Formatted at 2026-09-26 02:52:38 on dist-test-slave-ktng"
server_key: "5370188f04d6445e7f8ab3fd403fcd02"
server_key_iv: "c14d911197d6148821b5f590abf87dbc"
server_key_version: "encryptionkey@0"
I20260926 02:52:38.558372 20981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 02:52:38.562052 20996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:38.563678 20981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:52:38.563860 20981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "8bb76518363e4fef8a35b679a2c498a9"
format_stamp: "Formatted at 2026-09-26 02:52:38 on dist-test-slave-ktng"
server_key: "5370188f04d6445e7f8ab3fd403fcd02"
server_key_iv: "c14d911197d6148821b5f590abf87dbc"
server_key_version: "encryptionkey@0"
I20260926 02:52:38.564090 20981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:38.585726 20981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:38.590679 20981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:38.591001 20981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:38.592329 20981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:52:38.594164 20981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:38.594267 20981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:38.594396 20981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:38.594458 20981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:38.627283 20981 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:43589
I20260926 02:52:38.627440 21108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:43589 every 8 connection(s)
I20260926 02:52:38.628595 20981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:52:38.634187 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 20981
I20260926 02:52:38.634385 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:52:38.634929 20199 external_mini_cluster.cc:1428] Setting key 795a32a52efc6e7455a099d76a15e728
Sep 26 02:52:38 dist-test-slave-ktng krb5kdc[20738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391158, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:38.648933 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:38.631077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:46261 (local address 127.19.185.254:46757)
0926 02:52:38.631257 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:38.631261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:38.632517 (+ 1256us) server_negotiation.cc:407] Connection header received
0926 02:52:38.634443 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:38.634451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:38.634539 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:38.634665 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:38.636546 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:38.637405 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:38.638936 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:38.639728 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:38.639873 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:38.643372 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:38.643377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:38.643381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:38.643411 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:38.645820 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:38.646511 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:38.646517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:38.646520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:38.646655 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:38.647061 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:38.647068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:38.647071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:38.647407 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:38.647552 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:38.648344 (+ 792us) server_negotiation.cc:299] Negotiation successful
0926 02:52:38.648559 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:52:38.649905 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:38.631665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:46757 (local address 127.19.185.194:46261)
0926 02:52:38.632353 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:38.632393 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:38.634130 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:38.634894 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:38.634902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:38.635605 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:38.636370 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:38.636397 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:38.637556 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:38.637560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:38.638788 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:38.638799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:38.639923 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:38.639927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:38.640125 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:38.640807 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:38.640829 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:38.643088 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:38.645957 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:38.645967 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:38.645984 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:38.646391 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:38.646764 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:38.646770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:38.646773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:38.646951 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:38.647536 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:38.647544 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:38.647919 (+ 375us) client_negotiation.cc:769] Sending connection context
0926 02:52:38.648264 (+ 345us) client_negotiation.cc:240] Negotiation successful
0926 02:52:38.648785 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":250,"threads_started":1}
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
I20260926 02:52:38.651369 21109 heartbeater.cc:347] Connected to a master server at 127.19.185.254:46757
I20260926 02:52:38.651687 21109 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:38.652441 21109 heartbeater.cc:511] Master 127.19.185.254:46757 requested a full tablet report, sending...
I20260926 02:52:38.653872 20784 ts_manager.cc:194] Registered new tserver with Master: 8bb76518363e4fef8a35b679a2c498a9 (127.19.185.194:43589)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
I20260926 02:52:38.654845 20784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:46261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:52:38.678356 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46757
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:38.995810 21114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:38.996098 21114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:38.996279 21114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:39.003396 21114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:39.003499 21114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:39.003705 21114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:52:39.013248 21114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46757
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:39.014580 21114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:39.015964 21114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:39.029212 21120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:39.029143 21122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:39.029299 21119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:39.030520 21114 server_base.cc:1034] running on GCE node
I20260926 02:52:39.031122 21114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:39.032168 21114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:39.033411 21114 hybrid_clock.cc:648] HybridClock initialized: now 1790391159033351 us; error 50 us; skew 500 ppm
I20260926 02:52:39.033743 21114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:39 dist-test-slave-ktng krb5kdc[20738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391159, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:39.037184 21114 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:52:39.038734 21114 webserver.cc:460] Webserver started at http://127.19.185.195:40685/ using document root <none> and password file <none>
I20260926 02:52:39.039497 21114 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:39.039644 21114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:39.039969 21114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:39.042905 21114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "f5d96834381d451c98ec2ed73694e068"
format_stamp: "Formatted at 2026-09-26 02:52:39 on dist-test-slave-ktng"
server_key: "0d251dfd645ab6fac2feac27f2741a02"
server_key_iv: "04ee2633a1f2436349fc8894457dadf2"
server_key_version: "encryptionkey@0"
I20260926 02:52:39.043702 21114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "f5d96834381d451c98ec2ed73694e068"
format_stamp: "Formatted at 2026-09-26 02:52:39 on dist-test-slave-ktng"
server_key: "0d251dfd645ab6fac2feac27f2741a02"
server_key_iv: "04ee2633a1f2436349fc8894457dadf2"
server_key_version: "encryptionkey@0"
I20260926 02:52:39.048763 21114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 02:52:39.052520 21129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:39.054054 21114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:52:39.054286 21114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "f5d96834381d451c98ec2ed73694e068"
format_stamp: "Formatted at 2026-09-26 02:52:39 on dist-test-slave-ktng"
server_key: "0d251dfd645ab6fac2feac27f2741a02"
server_key_iv: "04ee2633a1f2436349fc8894457dadf2"
server_key_version: "encryptionkey@0"
I20260926 02:52:39.054539 21114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:39.072757 21114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:39.077739 21114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:39.078085 21114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:39.079126 21114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:52:39.080673 21114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:39.080786 21114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:39.080971 21114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:39.081069 21114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:39.114646 21114 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:42241
I20260926 02:52:39.114748 21241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:42241 every 8 connection(s)
I20260926 02:52:39.116040 21114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:52:39.124989 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 21114
I20260926 02:52:39.125196 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:52:39.125698 20199 external_mini_cluster.cc:1428] Setting key 270f37d74e709cd0e8d4860dd85e3028
Sep 26 02:52:39 dist-test-slave-ktng krb5kdc[20738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391159, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:39.136655 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.118651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:33865 (local address 127.19.185.254:46757)
0926 02:52:39.118836 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:39.118840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:39.119992 (+ 1152us) server_negotiation.cc:407] Connection header received
0926 02:52:39.121769 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:39.121775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:39.121866 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:39.121974 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:39.123464 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.124209 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.126416 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.126929 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.127061 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.130765 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:39.130769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:39.130773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:39.130803 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:39.133185 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.133872 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.133885 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.133887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.134006 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.134642 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.134648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.134651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.135058 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:39.135265 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:39.136066 (+ 801us) server_negotiation.cc:299] Negotiation successful
0926 02:52:39.136314 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:52:39.137327 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.119137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:46757 (local address 127.19.185.195:33865)
0926 02:52:39.119842 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:39.119878 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:39.121508 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:39.122196 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:39.122204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:39.122743 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:39.123260 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:39.123276 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.124366 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.124376 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:39.126281 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:39.126292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.127142 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.127152 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:39.127498 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.128200 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:39.128220 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:39.130561 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:39.133323 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:39.133332 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:39.133348 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:39.133748 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:39.134115 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:39.134121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:39.134124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:39.134471 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:39.135254 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:39.135305 (+ 51us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:39.135702 (+ 397us) client_negotiation.cc:769] Sending connection context
0926 02:52:39.135924 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 02:52:39.136281 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":289,"threads_started":1}
I20260926 02:52:39.138772 21242 heartbeater.cc:347] Connected to a master server at 127.19.185.254:46757
I20260926 02:52:39.139094 21242 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:39.139962 21242 heartbeater.cc:511] Master 127.19.185.254:46757 requested a full tablet report, sending...
I20260926 02:52:39.141530 20784 ts_manager.cc:194] Registered new tserver with Master: f5d96834381d451c98ec2ed73694e068 (127.19.185.195:42241)
I20260926 02:52:39.142331 20784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:33865
I20260926 02:52:39.153750 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:52:39.168519 20975 heartbeater.cc:503] Master 127.19.185.254:46757 was elected leader, sending a full tablet report...
I20260926 02:52:39.174515 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.160258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.19.185.254:46757)
0926 02:52:39.160483 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:39.160491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:39.161187 (+ 696us) server_negotiation.cc:407] Connection header received
0926 02:52:39.161394 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:39.161400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:39.161477 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:39.161587 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:39.163186 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.163992 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.165275 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.165829 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.165969 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.169161 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:39.169169 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:39.169175 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:39.169221 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:39.172384 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.172966 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.172970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.172972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.173089 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.173405 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.173408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.173410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.173707 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:39.173887 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:39.174135 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 02:52:39.174284 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 02:52:39.180738 20784 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:52:39.187666 20784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:52:39.224501 21256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.200514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:42241 (local address 127.0.0.1:32898)
0926 02:52:39.215595 (+ 15081us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:39.215622 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:39.215760 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:39.217296 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:39.217301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:39.217323 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:39.217763 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:39.217776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.219827 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.219832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:39.221751 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:39.221762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.223454 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.223461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:39.223650 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.223702 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 02:52:39.223817 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 02:52:39.224023 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14919,"mutex_wait_us":37,"thread_start_us":14770,"threads_started":1}
I20260926 02:52:39.225080 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.200425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.19.185.193:36311)
0926 02:52:39.214032 (+ 13607us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:39.214047 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:39.214085 (+ 38us) server_negotiation.cc:407] Connection header received
0926 02:52:39.214233 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:39.214246 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:39.214586 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:39.214831 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:39.217051 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.218755 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.221233 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.223659 (+ 2426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.223888 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.224067 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:39.224611 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 02:52:39.224773 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13375,"thread_start_us":15434,"threads_started":1}
I20260926 02:52:39.225175 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.215941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:43589 (local address 127.0.0.1:55126)
0926 02:52:39.217042 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:39.217065 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:39.217263 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:39.218081 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:39.218085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:39.218120 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:39.218475 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:39.218484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.220174 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.220182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:39.222204 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:39.222220 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.224282 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.224288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:39.224493 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.224532 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 02:52:39.224639 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 02:52:39.224762 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":60,"thread_start_us":1849,"threads_started":1}
I20260926 02:52:39.225400 21254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.200199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:36311 (local address 127.0.0.1:43184)
0926 02:52:39.212911 (+ 12712us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:39.212979 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:39.213238 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:39.216009 (+ 2771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:39.216019 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:39.216057 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:39.216810 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:39.216828 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.218886 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.218891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:39.221047 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:39.221061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.223888 (+ 2827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.223894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:39.224059 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.224097 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:52:39.224192 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 02:52:39.224325 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12326,"mutex_wait_us":27,"thread_start_us":12683,"threads_started":1}
I20260926 02:52:39.225667 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.200780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.19.185.195:42241)
0926 02:52:39.216689 (+ 15909us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:39.216701 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:39.216740 (+ 39us) server_negotiation.cc:407] Connection header received
0926 02:52:39.216833 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:39.216841 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:39.217085 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:39.218001 (+ 916us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:39.218202 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.219691 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.221903 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.223193 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.224248 (+ 1055us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.224389 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:39.224477 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 02:52:39.225011 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15680,"thread_start_us":15490,"threads_started":1}
I20260926 02:52:39.225824 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.214328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.19.185.194:43589)
0926 02:52:39.215872 (+ 1544us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:39.215880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:39.217160 (+ 1280us) server_negotiation.cc:407] Connection header received
0926 02:52:39.217456 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:39.217463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:39.217673 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:39.217829 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:39.218638 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.220039 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.222366 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.223833 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.223983 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.224143 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:39.225432 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0926 02:52:39.225591 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1399,"thread_start_us":1289,"threads_started":1}
I20260926 02:52:39.229583 20910 tablet_service.cc:1467] Processing CreateTablet for tablet 141d0f6906204e4eb54f6b30e6a1427e (DEFAULT_TABLE table=test-table [id=657df864a07448139f5d4bda64288454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:39.230218 21044 tablet_service.cc:1467] Processing CreateTablet for tablet 141d0f6906204e4eb54f6b30e6a1427e (DEFAULT_TABLE table=test-table [id=657df864a07448139f5d4bda64288454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:39.230217 21177 tablet_service.cc:1467] Processing CreateTablet for tablet 141d0f6906204e4eb54f6b30e6a1427e (DEFAULT_TABLE table=test-table [id=657df864a07448139f5d4bda64288454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:39.231602 20910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141d0f6906204e4eb54f6b30e6a1427e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:39.232002 21044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141d0f6906204e4eb54f6b30e6a1427e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:39.232002 21177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141d0f6906204e4eb54f6b30e6a1427e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:39.246263 21260 tablet_bootstrap.cc:492] T 141d0f6906204e4eb54f6b30e6a1427e P ce3dfcf5969e4396b009841436d66f22: Bootstrap starting.
I20260926 02:52:39.252796 21262 tablet_bootstrap.cc:492] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068: Bootstrap starting.
I20260926 02:52:39.253067 21260 tablet_bootstrap.cc:654] T 141d0f6906204e4eb54f6b30e6a1427e P ce3dfcf5969e4396b009841436d66f22: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:39.255057 21261 tablet_bootstrap.cc:492] T 141d0f6906204e4eb54f6b30e6a1427e P 8bb76518363e4fef8a35b679a2c498a9: Bootstrap starting.
I20260926 02:52:39.256070 21260 log.cc:824] T 141d0f6906204e4eb54f6b30e6a1427e P ce3dfcf5969e4396b009841436d66f22: Log is configured to *not* fsync() on all Append() calls
I20260926 02:52:39.258951 21262 tablet_bootstrap.cc:654] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:39.260458 21261 tablet_bootstrap.cc:654] T 141d0f6906204e4eb54f6b30e6a1427e P 8bb76518363e4fef8a35b679a2c498a9: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:39.261122 21262 log.cc:824] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068: Log is configured to *not* fsync() on all Append() calls
I20260926 02:52:39.262038 21260 tablet_bootstrap.cc:492] T 141d0f6906204e4eb54f6b30e6a1427e P ce3dfcf5969e4396b009841436d66f22: No bootstrap required, opened a new log
I20260926 02:52:39.262396 21261 log.cc:824] T 141d0f6906204e4eb54f6b30e6a1427e P 8bb76518363e4fef8a35b679a2c498a9: Log is configured to *not* fsync() on all Append() calls
I20260926 02:52:39.262557 21260 ts_tablet_manager.cc:1397] T 141d0f6906204e4eb54f6b30e6a1427e P ce3dfcf5969e4396b009841436d66f22: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260926 02:52:39.266116 21262 tablet_bootstrap.cc:492] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068: No bootstrap required, opened a new log
I20260926 02:52:39.266455 21262 ts_tablet_manager.cc:1397] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260926 02:52:39.267454 21261 tablet_bootstrap.cc:492] T 141d0f6906204e4eb54f6b30e6a1427e P 8bb76518363e4fef8a35b679a2c498a9: No bootstrap required, opened a new log
I20260926 02:52:39.268061 21261 ts_tablet_manager.cc:1397] T 141d0f6906204e4eb54f6b30e6a1427e P 8bb76518363e4fef8a35b679a2c498a9: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260926 02:52:39.276741 21262 raft_consensus.cc:348] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dfcf5969e4396b009841436d66f22" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36311 } } peers { permanent_uuid: "f5d96834381d451c98ec2ed73694e068" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42241 } } peers { permanent_uuid: "8bb76518363e4fef8a35b679a2c498a9" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 43589 } }
I20260926 02:52:39.277248 21262 raft_consensus.cc:374] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:39.277438 21262 raft_consensus.cc:729] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d96834381d451c98ec2ed73694e068, State: Initialized, Role: FOLLOWER
I20260926 02:52:39.278216 21262 consensus_queue.cc:260] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dfcf5969e4396b009841436d66f22" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36311 } } peers { permanent_uuid: "f5d96834381d451c98ec2ed73694e068" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42241 } } peers { permanent_uuid: "8bb76518363e4fef8a35b679a2c498a9" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 43589 } }
I20260926 02:52:39.280037 21242 heartbeater.cc:503] Master 127.19.185.254:46757 was elected leader, sending a full tablet report...
I20260926 02:52:39.280297 21260 raft_consensus.cc:348] T 141d0f6906204e4eb54f6b30e6a1427e P ce3dfcf5969e4396b009841436d66f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dfcf5969e4396b009841436d66f22" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36311 } } peers { permanent_uuid: "f5d96834381d451c98ec2ed73694e068" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42241 } } peers { permanent_uuid: "8bb76518363e4fef8a35b679a2c498a9" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 43589 } }
I20260926 02:52:39.280999 21260 raft_consensus.cc:374] T 141d0f6906204e4eb54f6b30e6a1427e P ce3dfcf5969e4396b009841436d66f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:39.281183 21260 raft_consensus.cc:729] T 141d0f6906204e4eb54f6b30e6a1427e P ce3dfcf5969e4396b009841436d66f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce3dfcf5969e4396b009841436d66f22, State: Initialized, Role: FOLLOWER
I20260926 02:52:39.282064 21262 ts_tablet_manager.cc:1428] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068: Time spent starting tablet: real 0.015s user 0.001s sys 0.015s
I20260926 02:52:39.282271 21260 consensus_queue.cc:260] T 141d0f6906204e4eb54f6b30e6a1427e P ce3dfcf5969e4396b009841436d66f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dfcf5969e4396b009841436d66f22" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36311 } } peers { permanent_uuid: "f5d96834381d451c98ec2ed73694e068" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42241 } } peers { permanent_uuid: "8bb76518363e4fef8a35b679a2c498a9" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 43589 } }
I20260926 02:52:39.285244 21260 ts_tablet_manager.cc:1428] T 141d0f6906204e4eb54f6b30e6a1427e P ce3dfcf5969e4396b009841436d66f22: Time spent starting tablet: real 0.022s user 0.015s sys 0.004s
I20260926 02:52:39.284906 21261 raft_consensus.cc:348] T 141d0f6906204e4eb54f6b30e6a1427e P 8bb76518363e4fef8a35b679a2c498a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dfcf5969e4396b009841436d66f22" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36311 } } peers { permanent_uuid: "f5d96834381d451c98ec2ed73694e068" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42241 } } peers { permanent_uuid: "8bb76518363e4fef8a35b679a2c498a9" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 43589 } }
I20260926 02:52:39.285526 21261 raft_consensus.cc:374] T 141d0f6906204e4eb54f6b30e6a1427e P 8bb76518363e4fef8a35b679a2c498a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:39.285718 21261 raft_consensus.cc:729] T 141d0f6906204e4eb54f6b30e6a1427e P 8bb76518363e4fef8a35b679a2c498a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb76518363e4fef8a35b679a2c498a9, State: Initialized, Role: FOLLOWER
I20260926 02:52:39.286500 21261 consensus_queue.cc:260] T 141d0f6906204e4eb54f6b30e6a1427e P 8bb76518363e4fef8a35b679a2c498a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dfcf5969e4396b009841436d66f22" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36311 } } peers { permanent_uuid: "f5d96834381d451c98ec2ed73694e068" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42241 } } peers { permanent_uuid: "8bb76518363e4fef8a35b679a2c498a9" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 43589 } }
I20260926 02:52:39.289325 21109 heartbeater.cc:503] Master 127.19.185.254:46757 was elected leader, sending a full tablet report...
I20260926 02:52:39.289989 21261 ts_tablet_manager.cc:1428] T 141d0f6906204e4eb54f6b30e6a1427e P 8bb76518363e4fef8a35b679a2c498a9: Time spent starting tablet: real 0.022s user 0.003s sys 0.017s
I20260926 02:52:39.333555 21266 raft_consensus.cc:482] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:52:39.333918 21266 raft_consensus.cc:504] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dfcf5969e4396b009841436d66f22" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36311 } } peers { permanent_uuid: "f5d96834381d451c98ec2ed73694e068" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42241 } } peers { permanent_uuid: "8bb76518363e4fef8a35b679a2c498a9" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 43589 } }
I20260926 02:52:39.336205 21266 leader_election.cc:288] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce3dfcf5969e4396b009841436d66f22 (127.19.185.193:36311), 8bb76518363e4fef8a35b679a2c498a9 (127.19.185.194:43589)
I20260926 02:52:39.345109 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.336589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:56421 (local address 127.19.185.193:36311)
0926 02:52:39.336885 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:39.336893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:39.336917 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:52:39.337038 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:39.337044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:39.337173 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:39.337291 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:39.338161 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.339772 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.342298 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.344062 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.344285 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.344366 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:39.344789 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 02:52:39.344879 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:52:39.345307 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.336433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:36311 (local address 127.19.185.195:56421)
0926 02:52:39.336690 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:39.336738 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:39.336884 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:39.337535 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:39.337541 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:39.337571 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:39.337963 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:39.337982 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.339935 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.339940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:39.342059 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:39.342071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.344270 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.344277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:39.344603 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.344651 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 02:52:39.344759 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 02:52:39.344913 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 02:52:39.346961 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.336819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:43589 (local address 127.19.185.195:32999)
0926 02:52:39.337956 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:39.337970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:39.338108 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:39.338424 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:39.338429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:39.338452 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:39.339400 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:39.339413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.341235 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.341239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:39.344156 (+ 2917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:39.344173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.346250 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.346254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:39.346398 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.346420 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 02:52:39.346494 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 02:52:39.346577 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"thread_start_us":931,"threads_started":1}
I20260926 02:52:39.347118 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.337441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:32999 (local address 127.19.185.194:43589)
0926 02:52:39.337731 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:39.337737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:39.338053 (+ 316us) server_negotiation.cc:407] Connection header received
0926 02:52:39.338241 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:39.338244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:39.338302 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:39.338371 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:39.339586 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.341027 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.344343 (+ 3316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.346009 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.346194 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.346276 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:39.346569 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 02:52:39.346719 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:52:39.346776 20930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141d0f6906204e4eb54f6b30e6a1427e" candidate_uuid: "f5d96834381d451c98ec2ed73694e068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce3dfcf5969e4396b009841436d66f22" is_pre_election: true
I20260926 02:52:39.347880 20930 raft_consensus.cc:2454] T 141d0f6906204e4eb54f6b30e6a1427e P ce3dfcf5969e4396b009841436d66f22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d96834381d451c98ec2ed73694e068 in term 0.
I20260926 02:52:39.348888 21132 leader_election.cc:302] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce3dfcf5969e4396b009841436d66f22, f5d96834381d451c98ec2ed73694e068; no voters:
I20260926 02:52:39.348749 21064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141d0f6906204e4eb54f6b30e6a1427e" candidate_uuid: "f5d96834381d451c98ec2ed73694e068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bb76518363e4fef8a35b679a2c498a9" is_pre_election: true
I20260926 02:52:39.349498 21266 raft_consensus.cc:2781] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:52:39.349592 21064 raft_consensus.cc:2454] T 141d0f6906204e4eb54f6b30e6a1427e P 8bb76518363e4fef8a35b679a2c498a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d96834381d451c98ec2ed73694e068 in term 0.
I20260926 02:52:39.349795 21266 raft_consensus.cc:482] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:52:39.349972 21266 raft_consensus.cc:3037] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:39.353152 21266 raft_consensus.cc:504] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dfcf5969e4396b009841436d66f22" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36311 } } peers { permanent_uuid: "f5d96834381d451c98ec2ed73694e068" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42241 } } peers { permanent_uuid: "8bb76518363e4fef8a35b679a2c498a9" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 43589 } }
I20260926 02:52:39.354058 21266 leader_election.cc:288] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [CANDIDATE]: Term 1 election: Requested vote from peers ce3dfcf5969e4396b009841436d66f22 (127.19.185.193:36311), 8bb76518363e4fef8a35b679a2c498a9 (127.19.185.194:43589)
I20260926 02:52:39.354516 20930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141d0f6906204e4eb54f6b30e6a1427e" candidate_uuid: "f5d96834381d451c98ec2ed73694e068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce3dfcf5969e4396b009841436d66f22"
I20260926 02:52:39.354735 21064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141d0f6906204e4eb54f6b30e6a1427e" candidate_uuid: "f5d96834381d451c98ec2ed73694e068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bb76518363e4fef8a35b679a2c498a9"
I20260926 02:52:39.354823 20930 raft_consensus.cc:3037] T 141d0f6906204e4eb54f6b30e6a1427e P ce3dfcf5969e4396b009841436d66f22 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:39.355046 21064 raft_consensus.cc:3037] T 141d0f6906204e4eb54f6b30e6a1427e P 8bb76518363e4fef8a35b679a2c498a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:39.358604 21064 raft_consensus.cc:2454] T 141d0f6906204e4eb54f6b30e6a1427e P 8bb76518363e4fef8a35b679a2c498a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d96834381d451c98ec2ed73694e068 in term 1.
I20260926 02:52:39.358829 20930 raft_consensus.cc:2454] T 141d0f6906204e4eb54f6b30e6a1427e P ce3dfcf5969e4396b009841436d66f22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d96834381d451c98ec2ed73694e068 in term 1.
I20260926 02:52:39.359212 21132 leader_election.cc:302] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bb76518363e4fef8a35b679a2c498a9, f5d96834381d451c98ec2ed73694e068; no voters:
I20260926 02:52:39.359598 21266 raft_consensus.cc:2781] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:39.360404 21266 raft_consensus.cc:686] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [term 1 LEADER]: Becoming Leader. State: Replica: f5d96834381d451c98ec2ed73694e068, State: Running, Role: LEADER
I20260926 02:52:39.360991 21266 consensus_queue.cc:237] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dfcf5969e4396b009841436d66f22" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36311 } } peers { permanent_uuid: "f5d96834381d451c98ec2ed73694e068" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42241 } } peers { permanent_uuid: "8bb76518363e4fef8a35b679a2c498a9" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 43589 } }
I20260926 02:52:39.366276 20783 catalog_manager.cc:5495] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5d96834381d451c98ec2ed73694e068 (127.19.185.195). New cstate: current_term: 1 leader_uuid: "f5d96834381d451c98ec2ed73694e068" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3dfcf5969e4396b009841436d66f22" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5d96834381d451c98ec2ed73694e068" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bb76518363e4fef8a35b679a2c498a9" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 43589 } health_report { overall_health: UNKNOWN } } }
W20260926 02:52:39.367614 21243 tablet.cc:2369] T 141d0f6906204e4eb54f6b30e6a1427e P f5d96834381d451c98ec2ed73694e068: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:52:39.380628 21110 tablet.cc:2369] T 141d0f6906204e4eb54f6b30e6a1427e P 8bb76518363e4fef8a35b679a2c498a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:52:39.388495 20976 tablet.cc:2369] T 141d0f6906204e4eb54f6b30e6a1427e P ce3dfcf5969e4396b009841436d66f22: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 02:52:39 dist-test-slave-ktng krb5kdc[20738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:52:39.412488 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.394571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.19.185.193:36311)
0926 02:52:39.394752 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:39.394756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:39.395900 (+ 1144us) server_negotiation.cc:407] Connection header received
0926 02:52:39.396156 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:39.396160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:39.396226 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:39.396332 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:39.398860 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.399943 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.401586 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.402173 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.402316 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.406657 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:39.406673 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:39.406689 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:39.406742 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:39.410032 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.410627 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.410634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.410637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.410752 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.411048 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.411051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.411053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.411479 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:39.411693 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:39.411946 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 02:52:39.412093 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 02:52:39 dist-test-slave-ktng krb5kdc[20738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391159, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:52:39 dist-test-slave-ktng krb5kdc[20738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391159, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:52:39.457669 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.443830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.19.185.193:36311)
0926 02:52:39.444038 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:39.444047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:39.444384 (+ 337us) server_negotiation.cc:407] Connection header received
0926 02:52:39.444598 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:39.444602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:39.444673 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:39.444785 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:39.446257 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.446978 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.448299 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.448688 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.448824 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.452568 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:39.452573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:39.452576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:39.452609 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:39.454955 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.455705 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.455710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.455714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.455840 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.456242 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.456247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.456250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.456538 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:39.456782 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:39.457083 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 02:52:39.457343 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 02:52:39 dist-test-slave-ktng krb5kdc[20738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:52:39 dist-test-slave-ktng krb5kdc[20738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:52:39.502923 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.489445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.19.185.193:36311)
0926 02:52:39.489676 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:39.489683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:39.489965 (+ 282us) server_negotiation.cc:407] Connection header received
0926 02:52:39.490180 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:39.490188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:39.490266 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:39.490371 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:39.491743 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.492415 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.493620 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.494058 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.494194 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.498104 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:39.498109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:39.498112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:39.498148 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:39.500536 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.501172 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.501178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.501184 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.501303 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.501655 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.501659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.501662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.501951 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:39.502196 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:39.502493 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 02:52:39.502675 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 02:52:39.504733 20890 tablet_service.cc:2908] Rejecting scan request for tablet 141d0f6906204e4eb54f6b30e6a1427e: Uninitialized: clean time has not yet been initialized
I20260926 02:52:39.507925 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 20847
I20260926 02:52:39.520665 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.508886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.19.185.193:36311)
0926 02:52:39.509100 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:39.509106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:39.509318 (+ 212us) server_negotiation.cc:407] Connection header received
0926 02:52:39.509509 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:39.509514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:39.509593 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:39.509701 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:39.510849 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.511609 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.512793 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.513202 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.513346 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.515627 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:39.515632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:39.515636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:39.515667 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:39.518024 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.518720 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.518727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.518731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.518851 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.519212 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.519219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.519224 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.519577 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:39.519871 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:39.520149 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 02:52:39.520357 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:52:39.521353 20970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:52:39.613741 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 20847
I20260926 02:52:39.630870 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 20981
Sep 26 02:52:39 dist-test-slave-ktng krb5kdc[20738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:52:39.645318 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.631930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.19.185.194:43589)
0926 02:52:39.632125 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:39.632129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:39.632146 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:52:39.632192 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:39.632195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:39.632252 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:39.632360 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:39.633550 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.634343 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.635775 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.636204 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.636339 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.640280 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:39.640292 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:39.640312 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:39.640352 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:39.642957 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.643643 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.643651 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.643653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.643787 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.644110 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.644114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.644117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.644454 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:39.644714 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:39.644914 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 02:52:39.645073 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:52:39.646015 21104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:52:39.733100 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 20981
I20260926 02:52:39.750418 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 21114
Sep 26 02:52:39 dist-test-slave-ktng krb5kdc[20738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:52:39.765713 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:39.751114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.19.185.195:42241)
0926 02:52:39.751599 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:39.751604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:39.751618 (+ 14us) server_negotiation.cc:407] Connection header received
0926 02:52:39.751677 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:39.751681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:39.751736 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:39.751861 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:39.753227 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.753940 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.755225 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:39.755735 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:39.755879 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:39.760144 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:39.760159 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:39.760175 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:39.760207 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:39.762948 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.763678 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.763685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.763688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.763818 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:39.764217 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:39.764221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:39.764223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:39.764571 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:39.764871 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:39.765100 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 02:52:39.765332 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":239,"threads_started":1}
I20260926 02:52:39.766386 21237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:52:39.856707 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 21114
I20260926 02:52:39.873914 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 20752
I20260926 02:52:39.875010 20815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:52:39.981288 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 20752
2026-09-26T02:52:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4154 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:52:40 dist-test-slave-ktng krb5kdc[21290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:52:40 dist-test-slave-ktng krb5kdc[21290](info): set up 2 sockets
Sep 26 02:52:40 dist-test-slave-ktng krb5kdc[21290](info): commencing operation
krb5kdc: starting...
W20260926 02:52:41.080230 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:52:41 dist-test-slave-ktng krb5kdc[21290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:52:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:52:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:52:41.267709 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:38783
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:38783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:41.580412 21303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:41.580695 21303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:41.580842 21303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:41.588068 21303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:52:41.588176 21303 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 02:52:41.588245 21303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:41.588361 21303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:41.588425 21303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:52:41.588541 21303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:52:41.598313 21303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:38783
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:38783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:41.599741 21303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:41.601060 21303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:41.613400 21308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:41.613308 21309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:41.613333 21311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:41.613667 21303 server_base.cc:1034] running on GCE node
I20260926 02:52:41.614741 21303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:41.616194 21303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:41.617486 21303 hybrid_clock.cc:648] HybridClock initialized: now 1790391161617419 us; error 43 us; skew 500 ppm
I20260926 02:52:41.617871 21303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:41 dist-test-slave-ktng krb5kdc[21290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391161, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:41.621094 21303 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:52:41.622490 21303 webserver.cc:460] Webserver started at http://127.19.185.254:46141/ using document root <none> and password file <none>
I20260926 02:52:41.623194 21303 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:41.623343 21303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:41.623721 21303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:41.626713 21303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "81ea2e5aea654b6f9cc822ed3de8d088"
format_stamp: "Formatted at 2026-09-26 02:52:41 on dist-test-slave-ktng"
server_key: "ad1235b93fc0717f5ad40fcb94a23bb1"
server_key_iv: "d02a96d53839c9126bf34e4b1d6bda1b"
server_key_version: "encryptionkey@0"
I20260926 02:52:41.627565 21303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "81ea2e5aea654b6f9cc822ed3de8d088"
format_stamp: "Formatted at 2026-09-26 02:52:41 on dist-test-slave-ktng"
server_key: "ad1235b93fc0717f5ad40fcb94a23bb1"
server_key_iv: "d02a96d53839c9126bf34e4b1d6bda1b"
server_key_version: "encryptionkey@0"
I20260926 02:52:41.632647 21303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:52:41.636111 21318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:41.637449 21303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:52:41.637635 21303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "81ea2e5aea654b6f9cc822ed3de8d088"
format_stamp: "Formatted at 2026-09-26 02:52:41 on dist-test-slave-ktng"
server_key: "ad1235b93fc0717f5ad40fcb94a23bb1"
server_key_iv: "d02a96d53839c9126bf34e4b1d6bda1b"
server_key_version: "encryptionkey@0"
I20260926 02:52:41.637859 21303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:41.657816 21303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:41.661890 21303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:41.662217 21303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:41.683852 21303 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:38783
I20260926 02:52:41.683878 21379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:38783 every 8 connection(s)
I20260926 02:52:41.685316 21303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:52:41.688944 21380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:41.694872 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 21303
I20260926 02:52:41.695137 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:52:41.695814 20199 external_mini_cluster.cc:1428] Setting key 87381f9315ea5b5570fe25e1be88119b
I20260926 02:52:41.699362 21380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088: Bootstrap starting.
I20260926 02:52:41.702828 21380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:41.704187 21380 log.cc:824] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088: Log is configured to *not* fsync() on all Append() calls
I20260926 02:52:41.706962 21380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088: No bootstrap required, opened a new log
Sep 26 02:52:41 dist-test-slave-ktng krb5kdc[21290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:41.714473 21380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ea2e5aea654b6f9cc822ed3de8d088" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38783 } }
I20260926 02:52:41.714871 21380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:41.714986 21380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ea2e5aea654b6f9cc822ed3de8d088, State: Initialized, Role: FOLLOWER
I20260926 02:52:41.715690 21380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ea2e5aea654b6f9cc822ed3de8d088" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38783 } }
I20260926 02:52:41.716003 21380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:52:41.716142 21380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:52:41.716329 21380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:41.719039 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:41.697881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.19.185.254:38783)
0926 02:52:41.698678 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:41.698689 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:41.698729 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:52:41.700557 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:41.700583 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:41.700986 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:41.701305 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:41.703093 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:41.705083 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:41.706319 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:41.706977 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:41.707181 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:41.710991 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:41.710999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:41.711014 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:41.711054 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:41.714225 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:41.714950 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:41.714959 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:41.714969 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:41.715180 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:41.715639 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:41.715644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:41.715649 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:41.716490 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:41.716778 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:41.717155 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 02:52:41.717624 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":382,"threads_started":1}
I20260926 02:52:41.719660 21380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ea2e5aea654b6f9cc822ed3de8d088" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38783 } }
I20260926 02:52:41.720240 21380 leader_election.cc:302] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81ea2e5aea654b6f9cc822ed3de8d088; no voters:
I20260926 02:52:41.720920 21380 leader_election.cc:288] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:52:41.721170 21385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:41.722149 21385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [term 1 LEADER]: Becoming Leader. State: Replica: 81ea2e5aea654b6f9cc822ed3de8d088, State: Running, Role: LEADER
I20260926 02:52:41.722787 21385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ea2e5aea654b6f9cc822ed3de8d088" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38783 } }
I20260926 02:52:41.723177 21380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:52:41.725916 21386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81ea2e5aea654b6f9cc822ed3de8d088" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ea2e5aea654b6f9cc822ed3de8d088" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38783 } } }
I20260926 02:52:41.726336 21386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [sys.catalog]: This master's current role is: LEADER
I20260926 02:52:41.726068 21387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81ea2e5aea654b6f9cc822ed3de8d088. Latest consensus state: current_term: 1 leader_uuid: "81ea2e5aea654b6f9cc822ed3de8d088" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ea2e5aea654b6f9cc822ed3de8d088" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38783 } } }
I20260926 02:52:41.726765 21387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088 [sys.catalog]: This master's current role is: LEADER
I20260926 02:52:41.731459 21392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:52:41.736943 21392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:52:41.746021 21392 catalog_manager.cc:1347] Generated new cluster ID: bb419d2ab86c49758e86d3f0c5d4f93d
I20260926 02:52:41.746131 21392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:52:41.767454 21392 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:52:41.769332 21392 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:52:41.785382 21392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088: Generated new TSK 0
I20260926 02:52:41.786279 21392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:52:41.843441 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38783
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:42.169392 21406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:42.169673 21406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:42.169832 21406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:42.177002 21406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:42.177109 21406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:42.177255 21406 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:52:42.177385 21406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:52:42.187088 21406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.185.254:38783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:42.188452 21406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:42.189853 21406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:42.202991 21412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:42.203012 21414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:42.203050 21411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:42.203780 21406 server_base.cc:1034] running on GCE node
I20260926 02:52:42.204381 21406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:42.205291 21406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:42.206529 21406 hybrid_clock.cc:648] HybridClock initialized: now 1790391162206484 us; error 48 us; skew 500 ppm
I20260926 02:52:42.206839 21406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:42 dist-test-slave-ktng krb5kdc[21290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391162, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:42.210049 21406 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:52:42.211468 21406 webserver.cc:460] Webserver started at http://127.19.185.193:44329/ using document root <none> and password file <none>
I20260926 02:52:42.212195 21406 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:42.212328 21406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:42.212646 21406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:42.215520 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "b7be1fe83ace40edb42e6be71657d77d"
format_stamp: "Formatted at 2026-09-26 02:52:42 on dist-test-slave-ktng"
server_key: "62ee29990c59d314c78b43629c5a750a"
server_key_iv: "46fc8d9e59dcda85243bdbce0852847d"
server_key_version: "encryptionkey@0"
I20260926 02:52:42.216307 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "b7be1fe83ace40edb42e6be71657d77d"
format_stamp: "Formatted at 2026-09-26 02:52:42 on dist-test-slave-ktng"
server_key: "62ee29990c59d314c78b43629c5a750a"
server_key_iv: "46fc8d9e59dcda85243bdbce0852847d"
server_key_version: "encryptionkey@0"
I20260926 02:52:42.221390 21406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 02:52:42.224926 21421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:42.226257 21406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:52:42.226430 21406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "b7be1fe83ace40edb42e6be71657d77d"
format_stamp: "Formatted at 2026-09-26 02:52:42 on dist-test-slave-ktng"
server_key: "62ee29990c59d314c78b43629c5a750a"
server_key_iv: "46fc8d9e59dcda85243bdbce0852847d"
server_key_version: "encryptionkey@0"
I20260926 02:52:42.226636 21406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:42.248327 21406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:42.252351 21406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:42.252630 21406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:42.255189 21406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:42.255316 21406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:42.255486 21406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:42.255563 21406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:52:42 dist-test-slave-ktng krb5kdc[21290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391162, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:42.280709 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:42.256029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:39181 (local address 127.19.185.254:38783)
0926 02:52:42.256587 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:42.256595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:42.257835 (+ 1240us) server_negotiation.cc:407] Connection header received
0926 02:52:42.260221 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:42.260227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:42.260324 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:42.260435 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:42.262786 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.264118 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.266580 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.267174 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.267340 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:42.272638 (+ 5298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:42.272643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:42.272647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:42.272681 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:42.276407 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:42.277614 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:42.277622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:42.277627 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:42.277813 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:42.278520 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:42.278526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:42.278529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:42.278820 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:42.278937 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:42.280224 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0926 02:52:42.280449 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":322,"threads_started":1}
I20260926 02:52:42.282646 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:42.256709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38783 (local address 127.19.185.193:39181)
0926 02:52:42.257642 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:42.257694 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:42.259782 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:42.260892 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:42.260908 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:42.261724 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:42.262560 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:42.262592 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.264331 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.264338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:42.266364 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:42.266380 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.267458 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.267466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:42.267687 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:42.268487 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:42.268533 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:42.272373 (+ 3840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:42.276640 (+ 4267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:42.276650 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:42.276672 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:42.277468 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:42.277967 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:42.277973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:42.277984 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:42.278322 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:42.278965 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:42.278977 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:42.279586 (+ 609us) client_negotiation.cc:769] Sending connection context
0926 02:52:42.280021 (+ 435us) client_negotiation.cc:240] Negotiation successful
0926 02:52:42.280764 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":421,"threads_started":1}
I20260926 02:52:42.306245 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:42.290442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38783 (local address 127.19.185.193:38329)
0926 02:52:42.292101 (+ 1659us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:42.292136 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:42.292256 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:42.292663 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:42.292670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:42.293304 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:42.293858 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:42.293869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.294848 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.294855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:42.296685 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:42.296698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.297723 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.297728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:42.297845 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:42.298599 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:42.298625 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:42.299493 (+ 868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:42.303264 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:42.303274 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:42.303280 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:42.303789 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:42.304267 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:42.304272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:42.304276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:42.304465 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:42.305118 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:42.305123 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:42.305285 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 02:52:42.305389 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 02:52:42.305721 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1501,"thread_start_us":1346,"threads_started":1}
I20260926 02:52:42.306286 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:42.290388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:38329 (local address 127.19.185.254:38783)
0926 02:52:42.290609 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:42.290618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:42.292233 (+ 1615us) server_negotiation.cc:407] Connection header received
0926 02:52:42.292380 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:42.292388 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:42.292484 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:42.292606 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:42.294010 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.294712 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.296841 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.297367 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.297524 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:42.299724 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:42.299729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:42.299732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:42.299761 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:42.303062 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:42.303959 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:42.303967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:42.303972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:42.304148 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:42.304621 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:42.304628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:42.304633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:42.304999 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:42.305156 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:42.305551 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 02:52:42.305818 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:52:42.307067 21406 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:35189
I20260926 02:52:42.307181 21541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:35189 every 8 connection(s)
I20260926 02:52:42.308486 21406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:52:42.310719 21542 heartbeater.cc:347] Connected to a master server at 127.19.185.254:38783
I20260926 02:52:42.310953 21542 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:42.311661 21542 heartbeater.cc:511] Master 127.19.185.254:38783 requested a full tablet report, sending...
I20260926 02:52:42.312039 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 21406
I20260926 02:52:42.312286 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:52:42.313024 20199 external_mini_cluster.cc:1428] Setting key 48c403b32673f93eeda16948b6705f20
I20260926 02:52:42.313725 21335 ts_manager.cc:194] Registered new tserver with Master: b7be1fe83ace40edb42e6be71657d77d (127.19.185.193:35189)
I20260926 02:52:42.315658 21335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:39181
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:52:42.354692 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38783
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:42.673436 21546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:42.673709 21546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:42.673879 21546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:42.681038 21546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:42.681214 21546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:42.681376 21546 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:52:42.681519 21546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:52:42.691583 21546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.185.254:38783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:42.693202 21546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:42.694480 21546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:42.707464 21551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:42.707551 21552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:42.707546 21554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:42.708599 21546 server_base.cc:1034] running on GCE node
I20260926 02:52:42.709240 21546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:42.710212 21546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:42.711433 21546 hybrid_clock.cc:648] HybridClock initialized: now 1790391162711371 us; error 36 us; skew 500 ppm
I20260926 02:52:42.711709 21546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:42 dist-test-slave-ktng krb5kdc[21290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391162, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:42.714848 21546 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:52:42.716365 21546 webserver.cc:460] Webserver started at http://127.19.185.194:42157/ using document root <none> and password file <none>
I20260926 02:52:42.717053 21546 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:42.717139 21546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:42.717406 21546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:42.720428 21546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "383a721b25b4481386c274bf05535136"
format_stamp: "Formatted at 2026-09-26 02:52:42 on dist-test-slave-ktng"
server_key: "ec34596ed98c71a3a95adf702411ee6b"
server_key_iv: "72edc44b46b5ab4776459fd3ffacb567"
server_key_version: "encryptionkey@0"
I20260926 02:52:42.721177 21546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "383a721b25b4481386c274bf05535136"
format_stamp: "Formatted at 2026-09-26 02:52:42 on dist-test-slave-ktng"
server_key: "ec34596ed98c71a3a95adf702411ee6b"
server_key_iv: "72edc44b46b5ab4776459fd3ffacb567"
server_key_version: "encryptionkey@0"
I20260926 02:52:42.726536 21546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:52:42.730125 21561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:42.731909 21546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:52:42.732077 21546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "383a721b25b4481386c274bf05535136"
format_stamp: "Formatted at 2026-09-26 02:52:42 on dist-test-slave-ktng"
server_key: "ec34596ed98c71a3a95adf702411ee6b"
server_key_iv: "72edc44b46b5ab4776459fd3ffacb567"
server_key_version: "encryptionkey@0"
I20260926 02:52:42.732321 21546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:42.749621 21546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:42.754014 21546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:42.754302 21546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:42.757179 21546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:42.757296 21546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:42.757434 21546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:42.757525 21546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:52:42 dist-test-slave-ktng krb5kdc[21290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391162, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:42.782588 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:42.757868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:45785 (local address 127.19.185.254:38783)
0926 02:52:42.758083 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:42.758089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:42.759610 (+ 1521us) server_negotiation.cc:407] Connection header received
0926 02:52:42.761885 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:42.761891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:42.761985 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:42.762100 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:42.764587 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.765668 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.768208 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.768780 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.768915 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:42.774836 (+ 5921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:42.774846 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:42.774858 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:42.774907 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:42.778291 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:42.779379 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:42.779387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:42.779390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:42.779553 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:42.780147 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:42.780154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:42.780158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:42.780573 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:42.780872 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:42.782004 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0926 02:52:42.782233 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:52:42.784240 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:42.758562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38783 (local address 127.19.185.194:45785)
0926 02:52:42.759437 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:42.759478 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:42.761512 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:42.762428 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:42.762444 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:42.763358 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:42.764374 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:42.764405 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.765824 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.765830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:42.768059 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:42.768070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.769268 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.769278 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:42.769564 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:42.770489 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:42.770515 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:42.774503 (+ 3988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:42.778454 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:42.778468 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:42.778491 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:42.779185 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:42.779708 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:42.779716 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:42.779720 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:42.780002 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:42.780980 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:42.780990 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:42.781585 (+ 595us) client_negotiation.cc:769] Sending connection context
0926 02:52:42.781907 (+ 322us) client_negotiation.cc:240] Negotiation successful
0926 02:52:42.782476 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":333,"threads_started":1}
I20260926 02:52:42.806090 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:42.790993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:36495 (local address 127.19.185.254:38783)
0926 02:52:42.791230 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:42.791234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:42.791867 (+ 633us) server_negotiation.cc:407] Connection header received
0926 02:52:42.792146 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:42.792150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:42.792224 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:42.792335 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:42.793288 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.793991 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.795851 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.796451 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.796527 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:42.799090 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:42.799094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:42.799098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:42.799129 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:42.802720 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:42.803782 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:42.803785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:42.803788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:42.803902 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:42.804414 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:42.804419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:42.804422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:42.804924 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:42.805076 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:42.805493 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 02:52:42.805718 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260926 02:52:42.806384 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:42.791098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38783 (local address 127.19.185.194:36495)
0926 02:52:42.791753 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:42.791771 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:42.791941 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:42.792359 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:42.792363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:42.792870 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:42.793144 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:42.793152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.794136 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.794146 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:42.795655 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:42.795675 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:42.796885 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:42.796895 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:42.797031 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:42.797783 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:42.797812 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:42.798885 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:42.802902 (+ 4017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:42.802912 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:42.802918 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:42.803609 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:42.804085 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:42.804092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:42.804100 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:42.804277 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:42.805059 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:42.805064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:42.805250 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 02:52:42.805395 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 02:52:42.805754 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":370,"threads_started":1}
I20260926 02:52:42.809650 21546 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:38081
I20260926 02:52:42.809672 21680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:38081 every 8 connection(s)
I20260926 02:52:42.811265 21546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:52:42.812080 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 21546
I20260926 02:52:42.812304 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:52:42.812876 20199 external_mini_cluster.cc:1428] Setting key c61e7344f3a65b898370f55a0e3bc441
I20260926 02:52:42.813898 21681 heartbeater.cc:347] Connected to a master server at 127.19.185.254:38783
I20260926 02:52:42.814167 21681 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:42.814988 21681 heartbeater.cc:511] Master 127.19.185.254:38783 requested a full tablet report, sending...
I20260926 02:52:42.816565 21335 ts_manager.cc:194] Registered new tserver with Master: 383a721b25b4481386c274bf05535136 (127.19.185.194:38081)
I20260926 02:52:42.817857 21335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:45785
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:52:42.854971 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38783
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:43.182495 21685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:43.182827 21685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:43.182998 21685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:43.190644 21685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:43.190802 21685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:43.190964 21685 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:52:43.191131 21685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:52:43.200894 21685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.185.254:38783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:43.202221 21685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:43.203523 21685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:43.217578 21690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:43.217625 21693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:43.217681 21691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:43.218348 21685 server_base.cc:1034] running on GCE node
I20260926 02:52:43.218964 21685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:43.219986 21685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:43.221235 21685 hybrid_clock.cc:648] HybridClock initialized: now 1790391163221169 us; error 43 us; skew 500 ppm
I20260926 02:52:43.221549 21685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:43 dist-test-slave-ktng krb5kdc[21290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391163, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:43.224922 21685 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:52:43.226403 21685 webserver.cc:460] Webserver started at http://127.19.185.195:35459/ using document root <none> and password file <none>
I20260926 02:52:43.227061 21685 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:43.227149 21685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:43.227483 21685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:43.230558 21685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "b2cc7dae405c40278c863a2cb55f8931"
format_stamp: "Formatted at 2026-09-26 02:52:43 on dist-test-slave-ktng"
server_key: "91bbb46e77d1ae993f0bfa239d93c24f"
server_key_iv: "ae0b52a8e8e4b8b698f285790bf93606"
server_key_version: "encryptionkey@0"
I20260926 02:52:43.231413 21685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "b2cc7dae405c40278c863a2cb55f8931"
format_stamp: "Formatted at 2026-09-26 02:52:43 on dist-test-slave-ktng"
server_key: "91bbb46e77d1ae993f0bfa239d93c24f"
server_key_iv: "ae0b52a8e8e4b8b698f285790bf93606"
server_key_version: "encryptionkey@0"
I20260926 02:52:43.239660 21685 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260926 02:52:43.244793 21700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:43.246399 21685 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:52:43.246600 21685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "b2cc7dae405c40278c863a2cb55f8931"
format_stamp: "Formatted at 2026-09-26 02:52:43 on dist-test-slave-ktng"
server_key: "91bbb46e77d1ae993f0bfa239d93c24f"
server_key_iv: "ae0b52a8e8e4b8b698f285790bf93606"
server_key_version: "encryptionkey@0"
I20260926 02:52:43.246847 21685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:43.289803 21685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:43.295027 21685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:43.295379 21685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:43.299074 21685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:43.299232 21685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 02:52:43.299422 21685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:43.299516 21685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:52:43 dist-test-slave-ktng krb5kdc[21290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391163, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:43.318320 21542 heartbeater.cc:503] Master 127.19.185.254:38783 was elected leader, sending a full tablet report...
I20260926 02:52:43.327507 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.299038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:40753 (local address 127.19.185.254:38783)
0926 02:52:43.299249 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:43.299256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:43.300862 (+ 1606us) server_negotiation.cc:407] Connection header received
0926 02:52:43.303286 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:43.303315 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:43.303407 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:43.303531 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:43.305727 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.308443 (+ 2716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.311716 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.312586 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.312743 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.318627 (+ 5884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:43.318636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:43.318641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:43.318682 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:43.322841 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:43.324015 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:43.324024 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:43.324028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:43.324205 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:43.324843 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:43.324848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:43.324851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:43.325252 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:43.325409 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:43.326737 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0926 02:52:43.327019 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:52:43.330760 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.299778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38783 (local address 127.19.185.195:40753)
0926 02:52:43.300698 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:43.300751 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:43.302894 (+ 2143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:43.303847 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:43.303867 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:43.304509 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:43.305487 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:43.305511 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.308689 (+ 3178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.308698 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:43.311402 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:43.311422 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.313095 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.313104 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:43.313410 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.314355 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:43.314394 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:43.318371 (+ 3977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:43.322987 (+ 4616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:43.322997 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:43.323017 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:43.323799 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:43.324373 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:43.324382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:43.324388 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:43.324683 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:43.325553 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:43.325565 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:43.326128 (+ 563us) client_negotiation.cc:769] Sending connection context
0926 02:52:43.328083 (+ 1955us) client_negotiation.cc:240] Negotiation successful
0926 02:52:43.328781 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":399,"threads_started":1}
I20260926 02:52:43.361132 21777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.340554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38783 (local address 127.19.185.195:53529)
0926 02:52:43.342083 (+ 1529us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:43.342105 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:43.342272 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:43.342765 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:43.342776 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:43.343626 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:43.344199 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:43.344221 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.345978 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.345984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:43.348417 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:43.348439 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.349532 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.349546 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:43.349794 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.350582 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:43.350613 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:43.352082 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:43.357121 (+ 5039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:43.357128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:43.357138 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:43.357873 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:43.358420 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:43.358428 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:43.358434 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:43.358641 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:43.359450 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:43.359455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:43.359741 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 02:52:43.359922 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 02:52:43.360388 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1348,"thread_start_us":1206,"threads_started":1}
I20260926 02:52:43.361640 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.340547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:53529 (local address 127.19.185.254:38783)
0926 02:52:43.340765 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:43.340773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:43.342221 (+ 1448us) server_negotiation.cc:407] Connection header received
0926 02:52:43.342422 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:43.342430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:43.342556 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:43.342666 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:43.344434 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.345768 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.348640 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.349297 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.349439 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.352387 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:43.352396 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:43.352404 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:43.352455 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:43.356941 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:43.358080 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:43.358088 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:43.358094 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:43.358269 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:43.358824 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:43.358831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:43.358836 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:43.359262 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:43.360634 (+ 1372us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:43.360815 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 02:52:43.361102 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:52:43.366654 21685 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:43325
I20260926 02:52:43.366742 21819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:43325 every 8 connection(s)
I20260926 02:52:43.367956 21685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:52:43.370110 21820 heartbeater.cc:347] Connected to a master server at 127.19.185.254:38783
I20260926 02:52:43.370378 21820 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:43.371245 21820 heartbeater.cc:511] Master 127.19.185.254:38783 requested a full tablet report, sending...
I20260926 02:52:43.372817 21335 ts_manager.cc:194] Registered new tserver with Master: b2cc7dae405c40278c863a2cb55f8931 (127.19.185.195:43325)
I20260926 02:52:43.373651 21335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:40753
I20260926 02:52:43.374493 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 21685
I20260926 02:52:43.374701 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:52:43.375389 20199 external_mini_cluster.cc:1428] Setting key bb919e445dfb84b31521d009b7b9e865
I20260926 02:52:43.378710 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:52:43 dist-test-slave-ktng krb5kdc[21290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391163, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:52:43 dist-test-slave-ktng krb5kdc[21290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391163, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:43.438215 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.420602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.19.185.254:38783)
0926 02:52:43.420836 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:43.420846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:43.421058 (+ 212us) server_negotiation.cc:407] Connection header received
0926 02:52:43.421224 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:43.421232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:43.421335 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:43.421474 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:43.422937 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.424387 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.425799 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.426751 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.426906 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.430889 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:43.430898 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:43.430905 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:43.430957 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:43.434976 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:43.435726 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:43.435734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:43.435740 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:43.435915 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:43.436402 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:43.436410 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:43.436415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:43.436857 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:43.437173 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:43.437363 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 02:52:43.437637 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 02:52:43.442947 21335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:52:43.450479 21335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:52:43.512174 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.463970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.19.185.194:38081)
0926 02:52:43.488350 (+ 24380us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:43.488364 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:43.501196 (+ 12832us) server_negotiation.cc:407] Connection header received
0926 02:52:43.501377 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:43.501389 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:43.502941 (+ 1552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:43.503147 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:43.503953 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.505467 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.509004 (+ 3537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.510452 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.510620 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.510804 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:43.511692 (+ 888us) server_negotiation.cc:299] Negotiation successful
0926 02:52:43.511860 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24139,"thread_start_us":24415,"threads_started":1}
I20260926 02:52:43.512231 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.463961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:35189 (local address 127.0.0.1:43342)
0926 02:52:43.502004 (+ 38043us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:43.502024 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:43.502146 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:43.503388 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:43.503393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:43.503424 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:43.503896 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:43.503910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.505721 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.505725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:43.508797 (+ 3072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:43.508806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.510966 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.510979 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:43.511271 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.511473 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 02:52:43.511612 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 02:52:43.511804 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37903,"mutex_wait_us":359,"thread_start_us":31703,"threads_started":1}
I20260926 02:52:43.513146 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.463715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.19.185.193:35189)
0926 02:52:43.498098 (+ 34383us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:43.498110 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:43.502282 (+ 4172us) server_negotiation.cc:407] Connection header received
0926 02:52:43.502397 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:43.502407 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:43.502685 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:43.502887 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:43.504102 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.505525 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.508963 (+ 3438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.510452 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.510627 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.510804 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:43.512684 (+ 1880us) server_negotiation.cc:299] Negotiation successful
0926 02:52:43.512845 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34151,"thread_start_us":37538,"threads_started":1}
I20260926 02:52:43.513418 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.463750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:38081 (local address 127.0.0.1:39284)
0926 02:52:43.501005 (+ 37255us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:43.501027 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:43.501161 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:43.503133 (+ 1972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:43.503138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:43.503162 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:43.503718 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:43.503737 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.505657 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.505663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:43.508762 (+ 3099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:43.508780 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.510996 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.511001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:43.511272 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.511473 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 02:52:43.512752 (+ 1279us) client_negotiation.cc:240] Negotiation successful
0926 02:52:43.512896 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37104,"mutex_wait_us":279,"thread_start_us":37957,"threads_started":1}
I20260926 02:52:43.516804 21611 tablet_service.cc:1467] Processing CreateTablet for tablet ecd16f7fab494b038b71d43ac139b555 (DEFAULT_TABLE table=test-table [id=0c170c28dfd546baa0b6b1cc49da1d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:43.517292 21472 tablet_service.cc:1467] Processing CreateTablet for tablet ecd16f7fab494b038b71d43ac139b555 (DEFAULT_TABLE table=test-table [id=0c170c28dfd546baa0b6b1cc49da1d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:43.518296 21611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd16f7fab494b038b71d43ac139b555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:43.518756 21472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd16f7fab494b038b71d43ac139b555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:43.523358 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.463385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:43325 (local address 127.0.0.1:35210)
0926 02:52:43.495107 (+ 31722us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:43.495175 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:43.495431 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:43.499857 (+ 4426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:43.499869 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:43.499910 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:43.500394 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:43.500410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.506683 (+ 6273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.506691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:43.509653 (+ 2962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:43.509673 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.512376 (+ 2703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.512382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:43.512613 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.512675 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 02:52:43.512771 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 02:52:43.512960 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31334,"mutex_wait_us":7,"thread_start_us":37332,"threads_started":1}
I20260926 02:52:43.524724 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.464028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.19.185.195:43325)
0926 02:52:43.498691 (+ 34663us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:43.498703 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:43.498742 (+ 39us) server_negotiation.cc:407] Connection header received
0926 02:52:43.498864 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:43.498875 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:43.499200 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:43.499450 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:43.503177 (+ 3727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.506496 (+ 3319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.509887 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.511806 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.512018 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.512218 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:43.524041 (+ 11823us) server_negotiation.cc:299] Negotiation successful
0926 02:52:43.524340 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34453,"thread_start_us":34267,"threads_started":1}
I20260926 02:52:43.528163 21750 tablet_service.cc:1467] Processing CreateTablet for tablet ecd16f7fab494b038b71d43ac139b555 (DEFAULT_TABLE table=test-table [id=0c170c28dfd546baa0b6b1cc49da1d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:43.529587 21750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd16f7fab494b038b71d43ac139b555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:43.543586 21840 tablet_bootstrap.cc:492] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d: Bootstrap starting.
I20260926 02:52:43.548038 21839 tablet_bootstrap.cc:492] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136: Bootstrap starting.
I20260926 02:52:43.551024 21840 tablet_bootstrap.cc:654] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:43.551622 21839 tablet_bootstrap.cc:654] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:43.552492 21840 log.cc:824] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:52:43.552865 21839 log.cc:824] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136: Log is configured to *not* fsync() on all Append() calls
I20260926 02:52:43.559834 21839 tablet_bootstrap.cc:492] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136: No bootstrap required, opened a new log
I20260926 02:52:43.560339 21839 ts_tablet_manager.cc:1397] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260926 02:52:43.564118 21841 tablet_bootstrap.cc:492] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931: Bootstrap starting.
I20260926 02:52:43.566159 21840 tablet_bootstrap.cc:492] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d: No bootstrap required, opened a new log
I20260926 02:52:43.566471 21840 ts_tablet_manager.cc:1397] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.013s
I20260926 02:52:43.567211 21841 tablet_bootstrap.cc:654] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:43.568221 21841 log.cc:824] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931: Log is configured to *not* fsync() on all Append() calls
I20260926 02:52:43.572317 21839 raft_consensus.cc:348] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } }
I20260926 02:52:43.572774 21839 raft_consensus.cc:374] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:43.572916 21839 raft_consensus.cc:729] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383a721b25b4481386c274bf05535136, State: Initialized, Role: FOLLOWER
I20260926 02:52:43.573722 21841 tablet_bootstrap.cc:492] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931: No bootstrap required, opened a new log
I20260926 02:52:43.573570 21839 consensus_queue.cc:260] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } }
I20260926 02:52:43.574004 21841 ts_tablet_manager.cc:1397] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260926 02:52:43.577883 21840 raft_consensus.cc:348] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } }
I20260926 02:52:43.578470 21840 raft_consensus.cc:374] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:43.578648 21840 raft_consensus.cc:729] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7be1fe83ace40edb42e6be71657d77d, State: Initialized, Role: FOLLOWER
I20260926 02:52:43.579210 21681 heartbeater.cc:503] Master 127.19.185.254:38783 was elected leader, sending a full tablet report...
I20260926 02:52:43.579993 21839 ts_tablet_manager.cc:1428] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260926 02:52:43.580524 21840 consensus_queue.cc:260] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } }
I20260926 02:52:43.581683 21841 raft_consensus.cc:348] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } }
I20260926 02:52:43.582190 21841 raft_consensus.cc:374] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:43.582317 21841 raft_consensus.cc:729] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2cc7dae405c40278c863a2cb55f8931, State: Initialized, Role: FOLLOWER
I20260926 02:52:43.583060 21841 consensus_queue.cc:260] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } }
I20260926 02:52:43.584607 21840 ts_tablet_manager.cc:1428] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260926 02:52:43.585634 21820 heartbeater.cc:503] Master 127.19.185.254:38783 was elected leader, sending a full tablet report...
I20260926 02:52:43.586357 21841 ts_tablet_manager.cc:1428] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
W20260926 02:52:43.620249 21821 tablet.cc:2369] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:52:43.634891 21845 raft_consensus.cc:482] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:52:43.635272 21845 raft_consensus.cc:504] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } }
I20260926 02:52:43.637034 21845 leader_election.cc:288] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7be1fe83ace40edb42e6be71657d77d (127.19.185.193:35189), b2cc7dae405c40278c863a2cb55f8931 (127.19.185.195:43325)
I20260926 02:52:43.644172 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.637311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:35189 (local address 127.19.185.194:59697)
0926 02:52:43.638146 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:43.638171 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:43.638338 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:43.639002 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:43.639008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:43.639038 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:43.639468 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:43.639476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.640431 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.640437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:43.641971 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:43.641984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.643223 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.643229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:43.643438 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.643461 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:52:43.643570 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 02:52:43.643719 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":2,"spinlock_wait_cycles":15744,"thread_start_us":591,"threads_started":1}
I20260926 02:52:43.644200 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.637528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:59697 (local address 127.19.185.193:35189)
0926 02:52:43.637752 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:43.637759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:43.638313 (+ 554us) server_negotiation.cc:407] Connection header received
0926 02:52:43.638804 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:43.638807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:43.638867 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:43.638952 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:43.639611 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.640295 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.642133 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.642971 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.643144 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.643233 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:43.643595 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 02:52:43.643730 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 02:52:43.644270 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.637444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:56875 (local address 127.19.185.195:43325)
0926 02:52:43.637652 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:43.637661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:43.638271 (+ 610us) server_negotiation.cc:407] Connection header received
0926 02:52:43.638465 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:43.638469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:43.638570 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:43.638653 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:43.639209 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.639977 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.642065 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.642981 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.643193 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.643272 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:43.643786 (+ 514us) server_negotiation.cc:299] Negotiation successful
0926 02:52:43.644041 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:52:43.644325 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.637345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:43325 (local address 127.19.185.194:56875)
0926 02:52:43.638177 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:43.638192 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:43.638338 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:43.638698 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:43.638701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:43.638727 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:43.639046 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:43.639058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.640102 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.640107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:43.641919 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:43.641931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.643251 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.643258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:43.643487 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.643510 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:52:43.643765 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 02:52:43.643904 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":91,"spinlock_wait_cycles":19072,"thread_start_us":383,"threads_started":1}
I20260926 02:52:43.645742 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd16f7fab494b038b71d43ac139b555" candidate_uuid: "383a721b25b4481386c274bf05535136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2cc7dae405c40278c863a2cb55f8931" is_pre_election: true
I20260926 02:52:43.645879 21492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd16f7fab494b038b71d43ac139b555" candidate_uuid: "383a721b25b4481386c274bf05535136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7be1fe83ace40edb42e6be71657d77d" is_pre_election: true
I20260926 02:52:43.646359 21492 raft_consensus.cc:2454] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 383a721b25b4481386c274bf05535136 in term 0.
I20260926 02:52:43.646361 21774 raft_consensus.cc:2454] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 383a721b25b4481386c274bf05535136 in term 0.
I20260926 02:52:43.646992 21563 leader_election.cc:302] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383a721b25b4481386c274bf05535136, b7be1fe83ace40edb42e6be71657d77d; no voters:
I20260926 02:52:43.647401 21845 raft_consensus.cc:2781] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:52:43.647552 21845 raft_consensus.cc:482] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:52:43.647658 21845 raft_consensus.cc:3037] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:43.649725 21845 raft_consensus.cc:504] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } }
I20260926 02:52:43.650347 21845 leader_election.cc:288] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [CANDIDATE]: Term 1 election: Requested vote from peers b7be1fe83ace40edb42e6be71657d77d (127.19.185.193:35189), b2cc7dae405c40278c863a2cb55f8931 (127.19.185.195:43325)
I20260926 02:52:43.650754 21492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd16f7fab494b038b71d43ac139b555" candidate_uuid: "383a721b25b4481386c274bf05535136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7be1fe83ace40edb42e6be71657d77d"
I20260926 02:52:43.650890 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd16f7fab494b038b71d43ac139b555" candidate_uuid: "383a721b25b4481386c274bf05535136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2cc7dae405c40278c863a2cb55f8931"
I20260926 02:52:43.650954 21492 raft_consensus.cc:3037] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:43.651098 21774 raft_consensus.cc:3037] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:43.653232 21774 raft_consensus.cc:2454] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 383a721b25b4481386c274bf05535136 in term 1.
I20260926 02:52:43.653239 21492 raft_consensus.cc:2454] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 383a721b25b4481386c274bf05535136 in term 1.
I20260926 02:52:43.653671 21562 leader_election.cc:302] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383a721b25b4481386c274bf05535136, b2cc7dae405c40278c863a2cb55f8931; no voters:
I20260926 02:52:43.653952 21845 raft_consensus.cc:2781] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:43.654310 21845 raft_consensus.cc:686] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [term 1 LEADER]: Becoming Leader. State: Replica: 383a721b25b4481386c274bf05535136, State: Running, Role: LEADER
I20260926 02:52:43.654830 21845 consensus_queue.cc:237] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } }
I20260926 02:52:43.659089 21334 catalog_manager.cc:5495] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 reported cstate change: term changed from 0 to 1, leader changed from <none> to 383a721b25b4481386c274bf05535136 (127.19.185.194). New cstate: current_term: 1 leader_uuid: "383a721b25b4481386c274bf05535136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } health_report { overall_health: HEALTHY } } }
I20260926 02:52:43.723891 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:43.718709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.19.185.194:38081)
0926 02:52:43.718882 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:43.718886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:43.718966 (+ 80us) server_negotiation.cc:407] Connection header received
0926 02:52:43.719223 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:43.719227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:43.719329 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:43.719440 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:52:43.719813 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.720521 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.721772 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:43.722147 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:43.722242 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:43.722661 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:52:43.723218 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:52:43.723372 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:43.723469 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 02:52:43.723531 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:52:43.731913 21774 raft_consensus.cc:1261] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931 [term 1 FOLLOWER]: Refusing update from remote peer 383a721b25b4481386c274bf05535136: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:43.731904 21492 raft_consensus.cc:1261] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d [term 1 FOLLOWER]: Refusing update from remote peer 383a721b25b4481386c274bf05535136: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:43.732847 21850 consensus_queue.cc:1035] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:43.733201 21845 consensus_queue.cc:1035] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:43.744726 21853 mvcc.cc:205] Tried to move back new op lower bound from 7333442206635704320 to 7333442206333927424. Current Snapshot: MvccSnapshot[applied={T|T < 7333442206635704320}]
I20260926 02:52:43.750895 21854 mvcc.cc:205] Tried to move back new op lower bound from 7333442206635704320 to 7333442206333927424. Current Snapshot: MvccSnapshot[applied={T|T < 7333442206635704320}]
I20260926 02:52:43.763975 21334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:52:43.764303 21334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:52:43.767843 21334 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088: Sending DeleteTablet for 3 replicas of tablet ecd16f7fab494b038b71d43ac139b555
I20260926 02:52:43.769076 21750 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd16f7fab494b038b71d43ac139b555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:43 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:35210
I20260926 02:52:43.769078 21472 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd16f7fab494b038b71d43ac139b555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:43 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:43342
I20260926 02:52:43.770074 21611 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd16f7fab494b038b71d43ac139b555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:43 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:39284
I20260926 02:52:43.770480 21334 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:52:43.771287 21334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:52:43.774168 21864 tablet_replica.cc:330] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136: stopping tablet replica
I20260926 02:52:43.780267 21864 raft_consensus.cc:2229] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:52:43.782284 21864 raft_consensus.cc:2258] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:43.783244 21749 tablet_service.cc:1467] Processing CreateTablet for tablet 702992427d7c4c89901e9401571d70e6 (DEFAULT_TABLE table=test-table [id=fdecb7b8490c412fa236515deee6aa34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:43.783342 21611 tablet_service.cc:1467] Processing CreateTablet for tablet 702992427d7c4c89901e9401571d70e6 (DEFAULT_TABLE table=test-table [id=fdecb7b8490c412fa236515deee6aa34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:43.783335 21471 tablet_service.cc:1467] Processing CreateTablet for tablet 702992427d7c4c89901e9401571d70e6 (DEFAULT_TABLE table=test-table [id=fdecb7b8490c412fa236515deee6aa34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:43.783798 21611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702992427d7c4c89901e9401571d70e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:43.783835 21749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702992427d7c4c89901e9401571d70e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:43.783864 21471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702992427d7c4c89901e9401571d70e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:43.790556 21840 tablet_bootstrap.cc:492] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d: Bootstrap starting.
I20260926 02:52:43.790848 21841 tablet_bootstrap.cc:492] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931: Bootstrap starting.
I20260926 02:52:43.792690 21840 tablet_bootstrap.cc:654] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:43.793139 21841 tablet_bootstrap.cc:654] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:43.793160 21839 tablet_bootstrap.cc:492] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136: Bootstrap starting.
I20260926 02:52:43.795177 21839 tablet_bootstrap.cc:654] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:43.803795 21840 tablet_bootstrap.cc:492] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d: No bootstrap required, opened a new log
I20260926 02:52:43.804030 21840 ts_tablet_manager.cc:1397] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20260926 02:52:43.804031 21864 ts_tablet_manager.cc:1905] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:43.804983 21840 raft_consensus.cc:348] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } }
I20260926 02:52:43.805310 21840 raft_consensus.cc:374] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:43.805436 21840 raft_consensus.cc:729] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7be1fe83ace40edb42e6be71657d77d, State: Initialized, Role: FOLLOWER
I20260926 02:52:43.805806 21840 consensus_queue.cc:260] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } }
I20260926 02:52:43.808712 21839 tablet_bootstrap.cc:492] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136: No bootstrap required, opened a new log
I20260926 02:52:43.808992 21839 ts_tablet_manager.cc:1397] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20260926 02:52:43.810410 21839 raft_consensus.cc:348] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } }
I20260926 02:52:43.810814 21839 raft_consensus.cc:374] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:43.810988 21839 raft_consensus.cc:729] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383a721b25b4481386c274bf05535136, State: Initialized, Role: FOLLOWER
I20260926 02:52:43.811825 21839 consensus_queue.cc:260] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } }
I20260926 02:52:43.812716 21839 ts_tablet_manager.cc:1428] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260926 02:52:43.813135 21840 ts_tablet_manager.cc:1428] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
W20260926 02:52:43.813699 21682 tablet.cc:2369] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:52:43.814749 21863 tablet_replica.cc:330] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931: stopping tablet replica
I20260926 02:52:43.815266 21863 raft_consensus.cc:2229] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:52:43.815846 21863 raft_consensus.cc:2258] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:43.816112 21841 tablet_bootstrap.cc:492] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931: No bootstrap required, opened a new log
I20260926 02:52:43.816243 21841 ts_tablet_manager.cc:1397] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.000s
I20260926 02:52:43.816641 21862 tablet_replica.cc:330] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d: stopping tablet replica
I20260926 02:52:43.817054 21841 raft_consensus.cc:348] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } }
I20260926 02:52:43.817217 21862 raft_consensus.cc:2229] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:52:43.817266 21841 raft_consensus.cc:374] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:43.817337 21841 raft_consensus.cc:729] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2cc7dae405c40278c863a2cb55f8931, State: Initialized, Role: FOLLOWER
I20260926 02:52:43.817662 21862 raft_consensus.cc:2258] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:43.820118 21863 ts_tablet_manager.cc:1905] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:43.820567 21864 ts_tablet_manager.cc:1918] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:52:43.820760 21864 log.cc:1196] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/ecd16f7fab494b038b71d43ac139b555
I20260926 02:52:43.821161 21841 consensus_queue.cc:260] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } }
I20260926 02:52:43.822218 21862 ts_tablet_manager.cc:1905] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:43.823091 21864 ts_tablet_manager.cc:1939] T ecd16f7fab494b038b71d43ac139b555 P 383a721b25b4481386c274bf05535136: Deleting consensus metadata
I20260926 02:52:43.825497 21322 catalog_manager.cc:4841] TS 383a721b25b4481386c274bf05535136 (127.19.185.194:38081): tablet ecd16f7fab494b038b71d43ac139b555 (table test-table [id=0c170c28dfd546baa0b6b1cc49da1d44]) successfully deleted
I20260926 02:52:43.826426 21841 ts_tablet_manager.cc:1428] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20260926 02:52:43.828433 21863 ts_tablet_manager.cc:1918] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:52:43.828651 21863 log.cc:1196] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/ecd16f7fab494b038b71d43ac139b555
I20260926 02:52:43.829145 21862 ts_tablet_manager.cc:1918] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:52:43.829247 21863 ts_tablet_manager.cc:1939] T ecd16f7fab494b038b71d43ac139b555 P b2cc7dae405c40278c863a2cb55f8931: Deleting consensus metadata
I20260926 02:52:43.829349 21862 log.cc:1196] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/ecd16f7fab494b038b71d43ac139b555
I20260926 02:52:43.829937 21862 ts_tablet_manager.cc:1939] T ecd16f7fab494b038b71d43ac139b555 P b7be1fe83ace40edb42e6be71657d77d: Deleting consensus metadata
I20260926 02:52:43.830945 21319 catalog_manager.cc:4841] TS b2cc7dae405c40278c863a2cb55f8931 (127.19.185.195:43325): tablet ecd16f7fab494b038b71d43ac139b555 (table test-table [id=0c170c28dfd546baa0b6b1cc49da1d44]) successfully deleted
I20260926 02:52:43.831735 21320 catalog_manager.cc:4841] TS b7be1fe83ace40edb42e6be71657d77d (127.19.185.193:35189): tablet ecd16f7fab494b038b71d43ac139b555 (table test-table [id=0c170c28dfd546baa0b6b1cc49da1d44]) successfully deleted
I20260926 02:52:44.024307 21847 raft_consensus.cc:482] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:52:44.024533 21847 raft_consensus.cc:504] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } }
I20260926 02:52:44.026041 21847 leader_election.cc:288] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7be1fe83ace40edb42e6be71657d77d (127.19.185.193:35189), 383a721b25b4481386c274bf05535136 (127.19.185.194:38081)
I20260926 02:52:44.034315 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.026912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:37721 (local address 127.19.185.194:38081)
0926 02:52:44.027103 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:44.027109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:44.027575 (+ 466us) server_negotiation.cc:407] Connection header received
0926 02:52:44.027790 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:44.027795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:44.027865 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:44.027943 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:44.028543 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.029658 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.032120 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.033092 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.033242 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.033297 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:44.033798 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 02:52:44.033929 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:52:44.034308 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.026415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:38081 (local address 127.19.185.195:37721)
0926 02:52:44.027491 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:44.027509 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:44.027657 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:44.027995 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:44.028000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:44.028040 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:44.028368 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:44.028379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.029809 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.029819 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:44.031947 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:44.031960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.033437 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.033441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:44.033574 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.033595 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 02:52:44.033772 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 02:52:44.033938 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":78,"thread_start_us":787,"threads_started":1}
I20260926 02:52:44.034555 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.026573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:50921 (local address 127.19.185.193:35189)
0926 02:52:44.026802 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:44.026811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:44.027328 (+ 517us) server_negotiation.cc:407] Connection header received
0926 02:52:44.027613 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:44.027620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:44.027704 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:44.027800 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:44.028271 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.029411 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.031971 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.033185 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.033360 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.033449 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:44.034103 (+ 654us) server_negotiation.cc:299] Negotiation successful
0926 02:52:44.034233 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:52:44.035120 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.026245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:35189 (local address 127.19.185.195:50921)
0926 02:52:44.027177 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:44.027203 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:44.027442 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:44.027818 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:44.027823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:44.027844 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:44.028111 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:44.028121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.029563 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.029569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:44.031785 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:44.031802 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.033695 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.033702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:44.033895 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.033926 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:52:44.034642 (+ 716us) client_negotiation.cc:240] Negotiation successful
0926 02:52:44.034781 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":26,"thread_start_us":693,"threads_started":1}
I20260926 02:52:44.035869 21492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702992427d7c4c89901e9401571d70e6" candidate_uuid: "b2cc7dae405c40278c863a2cb55f8931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7be1fe83ace40edb42e6be71657d77d" is_pre_election: true
I20260926 02:52:44.035928 21631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702992427d7c4c89901e9401571d70e6" candidate_uuid: "b2cc7dae405c40278c863a2cb55f8931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383a721b25b4481386c274bf05535136" is_pre_election: true
I20260926 02:52:44.036180 21492 raft_consensus.cc:2454] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2cc7dae405c40278c863a2cb55f8931 in term 0.
I20260926 02:52:44.036425 21631 raft_consensus.cc:2454] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2cc7dae405c40278c863a2cb55f8931 in term 0.
I20260926 02:52:44.036676 21702 leader_election.cc:302] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2cc7dae405c40278c863a2cb55f8931, b7be1fe83ace40edb42e6be71657d77d; no voters:
I20260926 02:52:44.037093 21847 raft_consensus.cc:2781] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:52:44.037217 21847 raft_consensus.cc:482] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:52:44.037321 21847 raft_consensus.cc:3037] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:44.039580 21847 raft_consensus.cc:504] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } }
I20260926 02:52:44.040205 21847 leader_election.cc:288] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [CANDIDATE]: Term 1 election: Requested vote from peers b7be1fe83ace40edb42e6be71657d77d (127.19.185.193:35189), 383a721b25b4481386c274bf05535136 (127.19.185.194:38081)
I20260926 02:52:44.040668 21492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702992427d7c4c89901e9401571d70e6" candidate_uuid: "b2cc7dae405c40278c863a2cb55f8931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7be1fe83ace40edb42e6be71657d77d"
I20260926 02:52:44.040757 21631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702992427d7c4c89901e9401571d70e6" candidate_uuid: "b2cc7dae405c40278c863a2cb55f8931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383a721b25b4481386c274bf05535136"
I20260926 02:52:44.040910 21492 raft_consensus.cc:3037] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:44.040957 21631 raft_consensus.cc:3037] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:44.044039 21492 raft_consensus.cc:2454] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2cc7dae405c40278c863a2cb55f8931 in term 1.
I20260926 02:52:44.044320 21631 raft_consensus.cc:2454] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2cc7dae405c40278c863a2cb55f8931 in term 1.
I20260926 02:52:44.044528 21702 leader_election.cc:302] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2cc7dae405c40278c863a2cb55f8931, b7be1fe83ace40edb42e6be71657d77d; no voters:
I20260926 02:52:44.044968 21847 raft_consensus.cc:2781] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:44.045480 21847 raft_consensus.cc:686] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [term 1 LEADER]: Becoming Leader. State: Replica: b2cc7dae405c40278c863a2cb55f8931, State: Running, Role: LEADER
I20260926 02:52:44.045873 21847 consensus_queue.cc:237] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } }
I20260926 02:52:44.049072 21334 catalog_manager.cc:5495] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2cc7dae405c40278c863a2cb55f8931 (127.19.185.195). New cstate: current_term: 1 leader_uuid: "b2cc7dae405c40278c863a2cb55f8931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } health_report { overall_health: HEALTHY } } }
W20260926 02:52:44.063582 21543 tablet.cc:2369] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 02:52:44 dist-test-slave-ktng krb5kdc[21290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391161, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:44.133148 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.109557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38783 (local address 127.0.0.1:54826)
0926 02:52:44.110582 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:44.110599 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:44.110721 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:44.111086 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:44.111089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:44.111800 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:44.111994 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:44.112002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.113495 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.113498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:44.114463 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:44.114474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.115440 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.115444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:44.115560 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.121971 (+ 6411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:44.121989 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:44.124713 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:44.129943 (+ 5230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:44.129954 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:44.129960 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:44.130407 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:44.130905 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:44.130908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:44.130911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:44.131100 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:44.132228 (+ 1128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:44.132235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:44.132379 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 02:52:44.132516 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 02:52:44.132855 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":747,"threads_started":1}
I20260926 02:52:44.133745 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.109633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.19.185.254:38783)
0926 02:52:44.110312 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:44.110318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:44.110706 (+ 388us) server_negotiation.cc:407] Connection header received
0926 02:52:44.110839 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:44.110844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:44.110922 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:44.111073 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:44.112165 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.113350 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.114610 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.115266 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.115430 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.124973 (+ 9543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:44.124980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:44.124985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:44.125031 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:44.129744 (+ 4713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:44.130541 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:44.130547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:44.130550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:44.130781 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:44.131280 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:44.131292 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:44.131318 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:44.131903 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:44.132090 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:44.132693 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 02:52:44.133062 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":478,"thread_start_us":305,"threads_started":1}
I20260926 02:52:44.140852 21334 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:54826:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 02:52:44.141975 21334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:52:44.149538 21750 tablet_service.cc:1467] Processing CreateTablet for tablet 572e382f3ff94b15b19b3772cdae9836 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c128622f2ee342608fbe87c8e1899804]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:52:44.149540 21611 tablet_service.cc:1467] Processing CreateTablet for tablet 572e382f3ff94b15b19b3772cdae9836 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c128622f2ee342608fbe87c8e1899804]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:52:44.150100 21611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572e382f3ff94b15b19b3772cdae9836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:44.150144 21750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572e382f3ff94b15b19b3772cdae9836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:44.157711 21472 tablet_service.cc:1467] Processing CreateTablet for tablet 572e382f3ff94b15b19b3772cdae9836 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c128622f2ee342608fbe87c8e1899804]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:52:44.158617 21472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572e382f3ff94b15b19b3772cdae9836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:44.159477 21841 tablet_bootstrap.cc:492] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931: Bootstrap starting.
I20260926 02:52:44.162150 21841 tablet_bootstrap.cc:654] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:44.163829 21839 tablet_bootstrap.cc:492] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136: Bootstrap starting.
I20260926 02:52:44.165963 21839 tablet_bootstrap.cc:654] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:44.168291 21840 tablet_bootstrap.cc:492] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d: Bootstrap starting.
I20260926 02:52:44.169981 21839 tablet_bootstrap.cc:492] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136: No bootstrap required, opened a new log
I20260926 02:52:44.170085 21840 tablet_bootstrap.cc:654] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:44.170173 21839 ts_tablet_manager.cc:1397] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260926 02:52:44.171020 21839 raft_consensus.cc:348] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } }
I20260926 02:52:44.171447 21839 raft_consensus.cc:374] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:44.171572 21839 raft_consensus.cc:729] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383a721b25b4481386c274bf05535136, State: Initialized, Role: FOLLOWER
I20260926 02:52:44.171912 21839 consensus_queue.cc:260] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } }
I20260926 02:52:44.172586 21845 tablet_replica.cc:439] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } }
I20260926 02:52:44.172967 21845 tablet_replica.cc:442] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:52:44.175844 21841 tablet_bootstrap.cc:492] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931: No bootstrap required, opened a new log
I20260926 02:52:44.175822 21839 ts_tablet_manager.cc:1428] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260926 02:52:44.176182 21841 ts_tablet_manager.cc:1397] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.008s
I20260926 02:52:44.177639 21841 raft_consensus.cc:348] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } }
I20260926 02:52:44.177973 21841 raft_consensus.cc:374] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:44.178117 21841 raft_consensus.cc:729] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2cc7dae405c40278c863a2cb55f8931, State: Initialized, Role: FOLLOWER
I20260926 02:52:44.178427 21841 consensus_queue.cc:260] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } }
I20260926 02:52:44.179085 21841 ts_tablet_manager.cc:1428] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 02:52:44.178978 21847 tablet_replica.cc:439] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } }
I20260926 02:52:44.179643 21847 tablet_replica.cc:442] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:52:44.184841 21840 tablet_bootstrap.cc:492] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d: No bootstrap required, opened a new log
I20260926 02:52:44.185020 21840 ts_tablet_manager.cc:1397] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.013s
I20260926 02:52:44.185794 21840 raft_consensus.cc:348] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } }
I20260926 02:52:44.186036 21840 raft_consensus.cc:374] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:44.186115 21840 raft_consensus.cc:729] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7be1fe83ace40edb42e6be71657d77d, State: Initialized, Role: FOLLOWER
I20260926 02:52:44.186369 21840 consensus_queue.cc:260] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } }
I20260926 02:52:44.187006 21840 ts_tablet_manager.cc:1428] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 02:52:44.186913 21846 tablet_replica.cc:439] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } }
I20260926 02:52:44.187392 21846 tablet_replica.cc:442] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:52:44.254789 21847 raft_consensus.cc:482] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:52:44.255049 21847 raft_consensus.cc:504] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } }
I20260926 02:52:44.255741 21847 leader_election.cc:288] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 383a721b25b4481386c274bf05535136 (127.19.185.194:38081), b7be1fe83ace40edb42e6be71657d77d (127.19.185.193:35189)
I20260926 02:52:44.256285 21631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572e382f3ff94b15b19b3772cdae9836" candidate_uuid: "b2cc7dae405c40278c863a2cb55f8931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383a721b25b4481386c274bf05535136" is_pre_election: true
I20260926 02:52:44.256335 21492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572e382f3ff94b15b19b3772cdae9836" candidate_uuid: "b2cc7dae405c40278c863a2cb55f8931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7be1fe83ace40edb42e6be71657d77d" is_pre_election: true
I20260926 02:52:44.256542 21631 raft_consensus.cc:2454] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2cc7dae405c40278c863a2cb55f8931 in term 0.
I20260926 02:52:44.256742 21492 raft_consensus.cc:2454] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2cc7dae405c40278c863a2cb55f8931 in term 0.
I20260926 02:52:44.257026 21704 leader_election.cc:302] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383a721b25b4481386c274bf05535136, b2cc7dae405c40278c863a2cb55f8931; no voters:
I20260926 02:52:44.257305 21847 raft_consensus.cc:2781] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:52:44.257393 21847 raft_consensus.cc:482] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:52:44.257477 21847 raft_consensus.cc:3037] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:44.259878 21847 raft_consensus.cc:504] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } }
I20260926 02:52:44.260390 21847 leader_election.cc:288] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [CANDIDATE]: Term 1 election: Requested vote from peers 383a721b25b4481386c274bf05535136 (127.19.185.194:38081), b7be1fe83ace40edb42e6be71657d77d (127.19.185.193:35189)
I20260926 02:52:44.260810 21492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572e382f3ff94b15b19b3772cdae9836" candidate_uuid: "b2cc7dae405c40278c863a2cb55f8931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7be1fe83ace40edb42e6be71657d77d"
I20260926 02:52:44.260869 21631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572e382f3ff94b15b19b3772cdae9836" candidate_uuid: "b2cc7dae405c40278c863a2cb55f8931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383a721b25b4481386c274bf05535136"
I20260926 02:52:44.261041 21492 raft_consensus.cc:3037] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:44.261162 21631 raft_consensus.cc:3037] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:44.262909 21492 raft_consensus.cc:2454] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2cc7dae405c40278c863a2cb55f8931 in term 1.
I20260926 02:52:44.263206 21631 raft_consensus.cc:2454] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2cc7dae405c40278c863a2cb55f8931 in term 1.
I20260926 02:52:44.263363 21702 leader_election.cc:302] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2cc7dae405c40278c863a2cb55f8931, b7be1fe83ace40edb42e6be71657d77d; no voters:
I20260926 02:52:44.263703 21847 raft_consensus.cc:2781] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:44.263881 21847 raft_consensus.cc:686] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [term 1 LEADER]: Becoming Leader. State: Replica: b2cc7dae405c40278c863a2cb55f8931, State: Running, Role: LEADER
I20260926 02:52:44.264154 21847 consensus_queue.cc:237] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } }
I20260926 02:52:44.265693 21870 tablet_replica.cc:439] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931: TxnStatusTablet state changed. Reason: New leader b2cc7dae405c40278c863a2cb55f8931. Latest consensus state: current_term: 1 leader_uuid: "b2cc7dae405c40278c863a2cb55f8931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } }
I20260926 02:52:44.265980 21870 tablet_replica.cc:442] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:52:44.266613 21883 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 02:52:44.266876 21883 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 02:52:44.267354 21335 catalog_manager.cc:5495] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2cc7dae405c40278c863a2cb55f8931 (127.19.185.195). New cstate: current_term: 1 leader_uuid: "b2cc7dae405c40278c863a2cb55f8931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } health_report { overall_health: UNKNOWN } } }
I20260926 02:52:44.296399 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.290187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.19.185.195:43325)
0926 02:52:44.290836 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:44.290841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:44.290876 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:52:44.290947 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:44.290951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:44.291020 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:44.291176 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:52:44.291619 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.292449 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.293916 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.294488 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.294634 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.295094 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:52:44.295708 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:52:44.295868 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:44.296017 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 02:52:44.296090 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":353,"threads_started":1}
I20260926 02:52:44.296492 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.290174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:43325 (local address 127.0.0.1:35216)
0926 02:52:44.290408 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:44.290430 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:44.290583 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:44.291207 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:44.291214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:44.291231 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:52:44.291461 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:44.291470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.292576 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.292584 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:44.293783 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:44.293794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.294653 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.294659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:44.294858 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.294908 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:52:44.295843 (+ 935us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:52:44.295867 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:52:44.295972 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 02:52:44.296152 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 02:52:44.300909 21492 raft_consensus.cc:1261] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d [term 1 FOLLOWER]: Refusing update from remote peer b2cc7dae405c40278c863a2cb55f8931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:44.301079 21631 raft_consensus.cc:1261] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136 [term 1 FOLLOWER]: Refusing update from remote peer b2cc7dae405c40278c863a2cb55f8931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:44.301685 21847 consensus_queue.cc:1035] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:44.302093 21870 consensus_queue.cc:1035] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:44.318625 21846 tablet_replica.cc:439] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d: TxnStatusTablet state changed. Reason: New leader b2cc7dae405c40278c863a2cb55f8931. Latest consensus state: current_term: 1 leader_uuid: "b2cc7dae405c40278c863a2cb55f8931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } }
I20260926 02:52:44.318939 21846 tablet_replica.cc:442] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:52:44.321995 21887 mvcc.cc:205] Tried to move back new op lower bound from 7333442208969383936 to 7333442208828432384. Current Snapshot: MvccSnapshot[applied={T|T < 7333442208969383936}]
I20260926 02:52:44.322638 21845 tablet_replica.cc:439] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136: TxnStatusTablet state changed. Reason: New leader b2cc7dae405c40278c863a2cb55f8931. Latest consensus state: current_term: 1 leader_uuid: "b2cc7dae405c40278c863a2cb55f8931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } }
I20260926 02:52:44.322953 21845 tablet_replica.cc:442] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:52:44.327361 21845 tablet_replica.cc:439] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2cc7dae405c40278c863a2cb55f8931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } }
I20260926 02:52:44.327643 21845 tablet_replica.cc:442] T 572e382f3ff94b15b19b3772cdae9836 P 383a721b25b4481386c274bf05535136: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:52:44.328604 21870 tablet_replica.cc:439] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2cc7dae405c40278c863a2cb55f8931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } }
I20260926 02:52:44.328948 21870 tablet_replica.cc:442] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:52:44.329223 21870 tablet_replica.cc:439] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2cc7dae405c40278c863a2cb55f8931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } }
I20260926 02:52:44.329479 21870 tablet_replica.cc:442] T 572e382f3ff94b15b19b3772cdae9836 P b2cc7dae405c40278c863a2cb55f8931: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:52:44.329962 21846 tablet_replica.cc:439] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2cc7dae405c40278c863a2cb55f8931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 } } peers { permanent_uuid: "b2cc7dae405c40278c863a2cb55f8931" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43325 } } peers { permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 } } }
I20260926 02:52:44.330185 21846 tablet_replica.cc:442] T 572e382f3ff94b15b19b3772cdae9836 P b7be1fe83ace40edb42e6be71657d77d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:52:44.342921 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.337629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.19.185.195:43325)
0926 02:52:44.337784 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:44.337793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:44.337870 (+ 77us) server_negotiation.cc:407] Connection header received
0926 02:52:44.338124 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:44.338129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:44.338207 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:44.338300 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:52:44.338787 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.339676 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.341055 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.341640 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.341754 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.342195 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:52:44.342345 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:52:44.342444 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:44.342557 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 02:52:44.342634 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:52:44.358810 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.353144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:32809 (local address 127.19.185.195:43325)
0926 02:52:44.353536 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:44.353546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:44.353937 (+ 391us) server_negotiation.cc:407] Connection header received
0926 02:52:44.354155 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:44.354166 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:44.354225 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:44.354311 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:52:44.354793 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.355616 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.357034 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.357448 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.357581 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.357995 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:52:44.358083 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:52:44.358182 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:44.358303 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 02:52:44.358425 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":247}
I20260926 02:52:44.359118 21900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.353208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:43325 (local address 127.19.185.195:32809)
0926 02:52:44.353847 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:44.353862 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:44.354011 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:44.354320 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:44.354325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:44.354335 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:52:44.354664 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:44.354672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.355743 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.355748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:44.356896 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:44.356907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.357623 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.357631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:44.357765 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.357814 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:52:44.358179 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:52:44.358189 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:52:44.358286 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 02:52:44.358461 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":23,"thread_start_us":342,"threads_started":1}
I20260926 02:52:44.375723 21630 raft_consensus.cc:1261] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136 [term 1 FOLLOWER]: Refusing update from remote peer b2cc7dae405c40278c863a2cb55f8931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:44.375912 21492 raft_consensus.cc:1261] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d [term 1 FOLLOWER]: Refusing update from remote peer b2cc7dae405c40278c863a2cb55f8931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:44.376394 21895 consensus_queue.cc:1035] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "383a721b25b4481386c274bf05535136" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:44.376821 21847 consensus_queue.cc:1035] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7be1fe83ace40edb42e6be71657d77d" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:44.447983 21334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:52:44.448305 21334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:52:44.451722 21334 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81ea2e5aea654b6f9cc822ed3de8d088: Sending DeleteTablet for 3 replicas of tablet 702992427d7c4c89901e9401571d70e6
I20260926 02:52:44.452612 21611 tablet_service.cc:1514] Processing DeleteTablet for tablet 702992427d7c4c89901e9401571d70e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:44 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:39284
I20260926 02:52:44.452584 21472 tablet_service.cc:1514] Processing DeleteTablet for tablet 702992427d7c4c89901e9401571d70e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:44 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:43342
I20260926 02:52:44.453307 21905 tablet_replica.cc:330] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136: stopping tablet replica
I20260926 02:52:44.453537 21905 raft_consensus.cc:2229] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:52:44.453824 21905 raft_consensus.cc:2258] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:44.455039 21905 ts_tablet_manager.cc:1905] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:44.455047 21750 tablet_service.cc:1514] Processing DeleteTablet for tablet 702992427d7c4c89901e9401571d70e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:44 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:35210
I20260926 02:52:44.459739 21904 tablet_replica.cc:330] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d: stopping tablet replica
I20260926 02:52:44.459967 21904 raft_consensus.cc:2229] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:52:44.460273 21904 raft_consensus.cc:2258] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:44.461526 21904 ts_tablet_manager.cc:1905] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:44.465744 21907 tablet_replica.cc:330] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931: stopping tablet replica
I20260926 02:52:44.466024 21907 raft_consensus.cc:2229] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:52:44.466794 21907 raft_consensus.cc:2258] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:44.468317 21907 ts_tablet_manager.cc:1905] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:44.471619 21905 ts_tablet_manager.cc:1918] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:52:44.471781 21905 log.cc:1196] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/702992427d7c4c89901e9401571d70e6
I20260926 02:52:44.472330 21905 ts_tablet_manager.cc:1939] T 702992427d7c4c89901e9401571d70e6 P 383a721b25b4481386c274bf05535136: Deleting consensus metadata
I20260926 02:52:44.473658 21322 catalog_manager.cc:4841] TS 383a721b25b4481386c274bf05535136 (127.19.185.194:38081): tablet 702992427d7c4c89901e9401571d70e6 (table test-table [id=fdecb7b8490c412fa236515deee6aa34]) successfully deleted
I20260926 02:52:44.475684 21904 ts_tablet_manager.cc:1918] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:52:44.475713 21907 ts_tablet_manager.cc:1918] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:52:44.475840 21904 log.cc:1196] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/702992427d7c4c89901e9401571d70e6
I20260926 02:52:44.475852 21907 log.cc:1196] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/702992427d7c4c89901e9401571d70e6
I20260926 02:52:44.476382 21907 ts_tablet_manager.cc:1939] T 702992427d7c4c89901e9401571d70e6 P b2cc7dae405c40278c863a2cb55f8931: Deleting consensus metadata
I20260926 02:52:44.476387 21904 ts_tablet_manager.cc:1939] T 702992427d7c4c89901e9401571d70e6 P b7be1fe83ace40edb42e6be71657d77d: Deleting consensus metadata
I20260926 02:52:44.477581 21320 catalog_manager.cc:4841] TS b7be1fe83ace40edb42e6be71657d77d (127.19.185.193:35189): tablet 702992427d7c4c89901e9401571d70e6 (table test-table [id=fdecb7b8490c412fa236515deee6aa34]) successfully deleted
I20260926 02:52:44.477881 21319 catalog_manager.cc:4841] TS b2cc7dae405c40278c863a2cb55f8931 (127.19.185.195:43325): tablet 702992427d7c4c89901e9401571d70e6 (table test-table [id=fdecb7b8490c412fa236515deee6aa34]) successfully deleted
Sep 26 02:52:44 dist-test-slave-ktng krb5kdc[21290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391163, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:52:44.494529 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.460870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.19.185.193:35189)
0926 02:52:44.461115 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:44.461125 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:44.472071 (+ 10946us) server_negotiation.cc:407] Connection header received
0926 02:52:44.472291 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:44.472297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:44.472379 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:44.472479 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:44.477877 (+ 5398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.478674 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.480415 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.481082 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.481276 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.486901 (+ 5625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:44.486923 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:44.486952 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:44.486992 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:44.491438 (+ 4446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:44.492238 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:44.492250 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:44.492255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:44.492441 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:44.492838 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:44.492844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:44.492848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:44.493352 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:44.493690 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:44.493865 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 02:52:44.494123 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 02:52:44.495420 21537 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43344
I20260926 02:52:44.514700 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.499110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.19.185.254:38783)
0926 02:52:44.499414 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:44.499421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:44.499660 (+ 239us) server_negotiation.cc:407] Connection header received
0926 02:52:44.499800 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:44.499804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:44.499876 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:44.499970 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:44.501890 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.502767 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.504838 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.505347 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.505454 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.508826 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:44.508832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:44.508835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:44.508874 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:44.511838 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:44.512810 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:44.512814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:44.512817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:44.512945 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:44.513372 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:44.513375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:44.513377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:44.513654 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:44.513834 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:44.514266 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 02:52:44.514443 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
W20260926 02:52:44.515393 21334 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54830
I20260926 02:52:44.517872 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 21406
I20260926 02:52:44.534186 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.519275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.19.185.193:35189)
0926 02:52:44.519497 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:44.519504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:44.519855 (+ 351us) server_negotiation.cc:407] Connection header received
0926 02:52:44.519970 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:44.519976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:44.520064 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:44.520189 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:44.521261 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.522456 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.523784 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.524540 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.524711 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.526846 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:44.526854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:44.526860 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:44.526905 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:44.530763 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:44.531803 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:44.531809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:44.531814 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:44.531972 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:44.532412 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:44.532418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:44.532422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:44.532817 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:44.533100 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:44.533504 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 02:52:44.533743 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260926 02:52:44.534950 21537 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43356
W20260926 02:52:44.535602 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 21406: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:52:44.535811 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 21406
I20260926 02:52:44.561509 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 21546
Sep 26 02:52:44 dist-test-slave-ktng krb5kdc[21290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391163, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:52:44.577207 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.562389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39304 (local address 127.19.185.194:38081)
0926 02:52:44.562893 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:44.562897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:44.562912 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:52:44.562963 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:44.562967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:44.563030 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:44.563151 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:44.564357 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.565079 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.566282 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.566691 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.566829 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.571168 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:44.571180 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:44.571196 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:44.571232 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:44.574088 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:44.574888 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:44.574896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:44.574898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:44.575133 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:44.575611 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:44.575616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:44.575618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:44.576011 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:44.576246 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:44.576611 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 02:52:44.576782 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":266,"threads_started":1}
W20260926 02:52:44.577944 21676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39304
W20260926 02:52:44.578619 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 21546: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:52:44.578807 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 21546
I20260926 02:52:44.600668 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 21685
Sep 26 02:52:44 dist-test-slave-ktng krb5kdc[21290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391163, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:52:44.617617 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:44.601521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.19.185.195:43325)
0926 02:52:44.601753 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:44.601760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:44.601783 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:52:44.601926 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:44.601932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:44.602014 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:44.602131 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:44.603351 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.604553 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.605860 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:44.606655 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:44.606796 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:44.610870 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:44.610884 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:44.610904 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:44.610960 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:44.614740 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:44.615499 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:44.615513 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:44.615516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:44.615648 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:44.616023 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:44.616029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:44.616032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:44.616412 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:44.616757 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:44.617044 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 02:52:44.617273 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260926 02:52:44.618278 21815 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234
W20260926 02:52:44.619017 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 21685: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:52:44.619180 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 21685
I20260926 02:52:44.639506 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 21303
I20260926 02:52:44.640734 21375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:52:44.754334 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 21303
2026-09-26T02:52:44Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4769 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:52:44 dist-test-slave-ktng krb5kdc[21916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:52:44 dist-test-slave-ktng krb5kdc[21916](info): set up 2 sockets
Sep 26 02:52:44 dist-test-slave-ktng krb5kdc[21916](info): commencing operation
krb5kdc: starting...
W20260926 02:52:45.876453 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:52:46 dist-test-slave-ktng krb5kdc[21916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:52:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:52:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:52:46.071640 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:42001
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:42001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:46.390268 21929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:46.390591 21929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:46.390771 21929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:46.397852 21929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:52:46.398051 21929 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 02:52:46.398146 21929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:46.398242 21929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:46.398334 21929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:52:46.398422 21929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:52:46.408237 21929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:42001
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:42001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:46.409654 21929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:46.411024 21929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:46.423079 21937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:46.423101 21935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:46.423336 21934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:46.423897 21929 server_base.cc:1034] running on GCE node
I20260926 02:52:46.424628 21929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:46.425962 21929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:46.427228 21929 hybrid_clock.cc:648] HybridClock initialized: now 1790391166427173 us; error 49 us; skew 500 ppm
I20260926 02:52:46.427587 21929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:46 dist-test-slave-ktng krb5kdc[21916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391166, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:46.430954 21929 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:52:46.432850 21929 webserver.cc:460] Webserver started at http://127.19.185.254:39489/ using document root <none> and password file <none>
I20260926 02:52:46.433598 21929 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:46.433725 21929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:46.434082 21929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:46.437158 21929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "56272cc72b814f7b908d32e53faa2d76"
format_stamp: "Formatted at 2026-09-26 02:52:46 on dist-test-slave-ktng"
server_key: "a46998b8dda2fe0f879c20b2836f1680"
server_key_iv: "dde5e72cb10fdf65a76a3c23c1b13e2d"
server_key_version: "encryptionkey@0"
I20260926 02:52:46.437976 21929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "56272cc72b814f7b908d32e53faa2d76"
format_stamp: "Formatted at 2026-09-26 02:52:46 on dist-test-slave-ktng"
server_key: "a46998b8dda2fe0f879c20b2836f1680"
server_key_iv: "dde5e72cb10fdf65a76a3c23c1b13e2d"
server_key_version: "encryptionkey@0"
I20260926 02:52:46.443362 21929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:52:46.446738 21944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:46.448192 21929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:52:46.448395 21929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "56272cc72b814f7b908d32e53faa2d76"
format_stamp: "Formatted at 2026-09-26 02:52:46 on dist-test-slave-ktng"
server_key: "a46998b8dda2fe0f879c20b2836f1680"
server_key_iv: "dde5e72cb10fdf65a76a3c23c1b13e2d"
server_key_version: "encryptionkey@0"
I20260926 02:52:46.448770 21929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:46.462859 21929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:46.467033 21929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:46.467347 21929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:46.491204 21929 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:42001
I20260926 02:52:46.491225 22005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:42001 every 8 connection(s)
I20260926 02:52:46.492923 21929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:52:46.496853 22006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:46.499974 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 21929
I20260926 02:52:46.500212 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:52:46.501003 20199 external_mini_cluster.cc:1428] Setting key 8e43b292f788d425adb60a98a9453caa
I20260926 02:52:46.510074 22006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76: Bootstrap starting.
I20260926 02:52:46.513722 22006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:46.514788 22006 log.cc:824] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:52:46 dist-test-slave-ktng krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:46.517819 22006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76: No bootstrap required, opened a new log
I20260926 02:52:46.524734 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:46.503142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40606 (local address 127.19.185.254:42001)
0926 02:52:46.504004 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:46.504020 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:46.504069 (+ 49us) server_negotiation.cc:407] Connection header received
0926 02:52:46.505915 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:46.505950 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:46.506492 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:46.506955 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:46.508427 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:46.510342 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:46.511769 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:46.512792 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:46.512973 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:46.517275 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:46.517283 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:46.517298 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:46.517332 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:46.520424 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:46.521200 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:46.521207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:46.521214 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:46.521371 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:46.521775 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:46.521778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:46.521780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:46.522328 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:46.522662 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:46.523158 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 02:52:46.523666 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":346,"threads_started":1}
I20260926 02:52:46.526226 22006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56272cc72b814f7b908d32e53faa2d76" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 42001 } }
I20260926 02:52:46.526690 22006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:46.526799 22006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56272cc72b814f7b908d32e53faa2d76, State: Initialized, Role: FOLLOWER
I20260926 02:52:46.527499 22006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56272cc72b814f7b908d32e53faa2d76" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 42001 } }
I20260926 02:52:46.527781 22006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:52:46.527918 22006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:52:46.528108 22006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:46.530408 22006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56272cc72b814f7b908d32e53faa2d76" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 42001 } }
I20260926 02:52:46.530900 22006 leader_election.cc:302] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56272cc72b814f7b908d32e53faa2d76; no voters:
I20260926 02:52:46.531527 22006 leader_election.cc:288] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:52:46.531623 22011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:46.532373 22011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [term 1 LEADER]: Becoming Leader. State: Replica: 56272cc72b814f7b908d32e53faa2d76, State: Running, Role: LEADER
I20260926 02:52:46.532927 22011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56272cc72b814f7b908d32e53faa2d76" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 42001 } }
I20260926 02:52:46.533617 22006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:52:46.535972 22012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56272cc72b814f7b908d32e53faa2d76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56272cc72b814f7b908d32e53faa2d76" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 42001 } } }
I20260926 02:52:46.535964 22013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56272cc72b814f7b908d32e53faa2d76. Latest consensus state: current_term: 1 leader_uuid: "56272cc72b814f7b908d32e53faa2d76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56272cc72b814f7b908d32e53faa2d76" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 42001 } } }
I20260926 02:52:46.536399 22012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [sys.catalog]: This master's current role is: LEADER
I20260926 02:52:46.536417 22013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76 [sys.catalog]: This master's current role is: LEADER
I20260926 02:52:46.538570 22017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:52:46.543603 22017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:52:46.553390 22017 catalog_manager.cc:1347] Generated new cluster ID: 95653ea48d6246b69c0e9e3f002c98f4
I20260926 02:52:46.553515 22017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:52:46.578601 22017 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:52:46.580556 22017 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:52:46.593616 22017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76: Generated new TSK 0
I20260926 02:52:46.594621 22017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:52:46.650993 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:42001
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:46.975273 22032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:46.975597 22032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:46.975701 22032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:46.983065 22032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:46.983186 22032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:46.983275 22032 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:52:46.983398 22032 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:52:46.983487 22032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:52:46.993302 22032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.185.254:42001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:46.994642 22032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:46.995949 22032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:47.009512 22037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:47.009490 22040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:47.009450 22038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:47.010447 22032 server_base.cc:1034] running on GCE node
I20260926 02:52:47.011124 22032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:47.012161 22032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:47.013398 22032 hybrid_clock.cc:648] HybridClock initialized: now 1790391167013334 us; error 58 us; skew 500 ppm
I20260926 02:52:47.013774 22032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:47 dist-test-slave-ktng krb5kdc[21916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391167, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:47.017086 22032 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:52:47.018659 22032 webserver.cc:460] Webserver started at http://127.19.185.193:41737/ using document root <none> and password file <none>
I20260926 02:52:47.019395 22032 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:47.019502 22032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:47.019761 22032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:47.022680 22032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "ed52fbfc571b4439a2683dbbb31dcd9f"
format_stamp: "Formatted at 2026-09-26 02:52:47 on dist-test-slave-ktng"
server_key: "6751bdf5bb93e39b10cd355347a7feb4"
server_key_iv: "0bdb18ced4447b35cac37194bac21226"
server_key_version: "encryptionkey@0"
I20260926 02:52:47.023488 22032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "ed52fbfc571b4439a2683dbbb31dcd9f"
format_stamp: "Formatted at 2026-09-26 02:52:47 on dist-test-slave-ktng"
server_key: "6751bdf5bb93e39b10cd355347a7feb4"
server_key_iv: "0bdb18ced4447b35cac37194bac21226"
server_key_version: "encryptionkey@0"
I20260926 02:52:47.028861 22032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:52:47.032318 22047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:47.033783 22032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:52:47.033968 22032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "ed52fbfc571b4439a2683dbbb31dcd9f"
format_stamp: "Formatted at 2026-09-26 02:52:47 on dist-test-slave-ktng"
server_key: "6751bdf5bb93e39b10cd355347a7feb4"
server_key_iv: "0bdb18ced4447b35cac37194bac21226"
server_key_version: "encryptionkey@0"
I20260926 02:52:47.034168 22032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:47.054699 22032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:47.058744 22032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:47.059077 22032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:47.061873 22032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:47.062007 22032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:47.062126 22032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:47.062220 22032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:52:47 dist-test-slave-ktng krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391167, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:47.086256 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:47.062806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:53569 (local address 127.19.185.254:42001)
0926 02:52:47.063336 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:47.063344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:47.064491 (+ 1147us) server_negotiation.cc:407] Connection header received
0926 02:52:47.066510 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:47.066517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:47.066603 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:47.066744 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:47.069157 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.070207 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.072757 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.073512 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.073672 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:47.078626 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:47.078631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:47.078635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:47.078668 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:47.082583 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:47.083400 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:47.083409 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:47.083416 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:47.083605 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:47.084071 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:47.084079 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:47.084085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:47.084535 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:47.084722 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:47.085412 (+ 690us) server_negotiation.cc:299] Negotiation successful
0926 02:52:47.085706 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":277,"threads_started":1}
I20260926 02:52:47.086932 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:47.063487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:42001 (local address 127.19.185.193:53569)
0926 02:52:47.064333 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:47.064370 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:47.066235 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:47.067025 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:47.067042 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:47.067988 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:47.068881 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:47.068911 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.070349 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.070364 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:47.072592 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:47.072612 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.073803 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.073814 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:47.074015 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:47.074838 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:47.074861 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:47.078356 (+ 3495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:47.082759 (+ 4403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:47.082769 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:47.082791 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:47.083227 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:47.083732 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:47.083736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:47.083738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:47.083948 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:47.084678 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:47.084684 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:47.085027 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 02:52:47.085258 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 02:52:47.085667 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":373,"threads_started":1}
I20260926 02:52:47.107653 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:47.093193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:60095 (local address 127.19.185.254:42001)
0926 02:52:47.093438 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:47.093446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:47.094140 (+ 694us) server_negotiation.cc:407] Connection header received
0926 02:52:47.094341 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:47.094346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:47.094416 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:47.094514 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:47.095815 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.096710 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.098025 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.098419 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.098543 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:47.101087 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:47.101095 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:47.101102 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:47.101145 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:47.104865 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:47.105566 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:47.105574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:47.105578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:47.105712 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:47.106175 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:47.106182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:47.106188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:47.106494 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:47.106652 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:47.107064 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 02:52:47.107283 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 02:52:47.107816 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:47.093259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:42001 (local address 127.19.185.193:60095)
0926 02:52:47.094016 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:47.094041 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:47.094203 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:47.094542 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:47.094546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:47.095240 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:47.095671 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:47.095682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.096847 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.096853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:47.097850 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:47.097862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.098633 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.098647 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:47.098811 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:47.099555 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:47.099574 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:47.100791 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:47.105014 (+ 4223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:47.105022 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:47.105026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:47.105444 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:47.105823 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:47.105828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:47.105832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:47.105966 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:47.106624 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:47.106628 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:47.106819 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 02:52:47.106964 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 02:52:47.107279 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"spinlock_wait_cycles":67840,"thread_start_us":373,"threads_started":1}
I20260926 02:52:47.117256 22032 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:36921
I20260926 02:52:47.117298 22167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:36921 every 8 connection(s)
I20260926 02:52:47.118464 22032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:52:47.119022 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 22032
I20260926 02:52:47.119253 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:52:47.120078 20199 external_mini_cluster.cc:1428] Setting key 4d7b97df91b9c9b13ae71f796d8dd49e
I20260926 02:52:47.120707 22168 heartbeater.cc:347] Connected to a master server at 127.19.185.254:42001
I20260926 02:52:47.120931 22168 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:47.121573 22168 heartbeater.cc:511] Master 127.19.185.254:42001 requested a full tablet report, sending...
I20260926 02:52:47.123493 21961 ts_manager.cc:194] Registered new tserver with Master: ed52fbfc571b4439a2683dbbb31dcd9f (127.19.185.193:36921)
I20260926 02:52:47.124883 21961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:53569
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:52:47.166635 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:42001
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:47.489873 22172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:47.490219 22172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:47.490409 22172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:47.501885 22172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:47.502022 22172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:47.502163 22172 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:52:47.502229 22172 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:52:47.502327 22172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:52:47.516655 22172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.185.254:42001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:47.517892 22172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:47.519169 22172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:47.532635 22180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:47.532716 22177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:47.532713 22178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:47.537074 22172 server_base.cc:1034] running on GCE node
I20260926 02:52:47.537955 22172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:47.539134 22172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:47.540374 22172 hybrid_clock.cc:648] HybridClock initialized: now 1790391167540322 us; error 71 us; skew 500 ppm
I20260926 02:52:47.540723 22172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:47 dist-test-slave-ktng krb5kdc[21916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391167, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:47.544195 22172 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:52:47.545732 22172 webserver.cc:460] Webserver started at http://127.19.185.194:42305/ using document root <none> and password file <none>
I20260926 02:52:47.546475 22172 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:47.546623 22172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:47.546940 22172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:47.550041 22172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "f265e515f6744cde815a3ca1b4932376"
format_stamp: "Formatted at 2026-09-26 02:52:47 on dist-test-slave-ktng"
server_key: "48acfb9549cca839089e7b73744eaa14"
server_key_iv: "34866fbbc709cb3edac85f6b56e24c79"
server_key_version: "encryptionkey@0"
I20260926 02:52:47.551112 22172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "f265e515f6744cde815a3ca1b4932376"
format_stamp: "Formatted at 2026-09-26 02:52:47 on dist-test-slave-ktng"
server_key: "48acfb9549cca839089e7b73744eaa14"
server_key_iv: "34866fbbc709cb3edac85f6b56e24c79"
server_key_version: "encryptionkey@0"
I20260926 02:52:47.556922 22172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 02:52:47.560516 22187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:47.561914 22172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:52:47.562103 22172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "f265e515f6744cde815a3ca1b4932376"
format_stamp: "Formatted at 2026-09-26 02:52:47 on dist-test-slave-ktng"
server_key: "48acfb9549cca839089e7b73744eaa14"
server_key_iv: "34866fbbc709cb3edac85f6b56e24c79"
server_key_version: "encryptionkey@0"
I20260926 02:52:47.562322 22172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:47.577755 22172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:47.582170 22172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:47.582468 22172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:47.585351 22172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:47.585472 22172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:47.585600 22172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:47.585700 22172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:52:47 dist-test-slave-ktng krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391167, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:47.610358 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:47.585857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:45987 (local address 127.19.185.254:42001)
0926 02:52:47.586112 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:47.586122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:47.587773 (+ 1651us) server_negotiation.cc:407] Connection header received
0926 02:52:47.590380 (+ 2607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:47.590396 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:47.590496 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:47.590627 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:47.593243 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.594457 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.596981 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.597672 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.597755 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:47.603609 (+ 5854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:47.603615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:47.603618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:47.603653 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:47.606425 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:47.607203 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:47.607211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:47.607220 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:47.607408 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:47.608110 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:47.608116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:47.608119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:47.608513 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:47.608634 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:47.609742 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0926 02:52:47.609987 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 02:52:47.611660 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:47.586564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:42001 (local address 127.19.185.194:45987)
0926 02:52:47.587601 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:47.587644 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:47.590029 (+ 2385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:47.590909 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:47.590925 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:47.592028 (+ 1103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:47.592993 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:47.593028 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.594617 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.594623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:47.596745 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:47.596769 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.598074 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.598079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:47.598302 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:47.599268 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:47.599354 (+ 86us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:47.603276 (+ 3922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:47.606587 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:47.606597 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:47.606615 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:47.607065 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:47.607544 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:47.607550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:47.607557 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:47.607917 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:47.608757 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:47.608770 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:47.609321 (+ 551us) client_negotiation.cc:769] Sending connection context
0926 02:52:47.609632 (+ 311us) client_negotiation.cc:240] Negotiation successful
0926 02:52:47.610245 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":377,"threads_started":1}
I20260926 02:52:47.635187 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:47.619187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:53235 (local address 127.19.185.254:42001)
0926 02:52:47.619409 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:47.619419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:47.620582 (+ 1163us) server_negotiation.cc:407] Connection header received
0926 02:52:47.620808 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:47.620812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:47.620874 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:47.620982 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:47.621974 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.622760 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.624620 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.625374 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.625545 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:47.628141 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:47.628146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:47.628149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:47.628180 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:47.632132 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:47.632847 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:47.632854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:47.632859 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:47.633034 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:47.633582 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:47.633586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:47.633589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:47.633986 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:47.634152 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:47.634590 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 02:52:47.634818 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:52:47.635352 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:47.619198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:42001 (local address 127.19.185.194:53235)
0926 02:52:47.620473 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:47.620492 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:47.620662 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:47.621014 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:47.621019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:47.621498 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:47.621790 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:47.621799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.622908 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.622914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:47.624328 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:47.624374 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:47.625641 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:47.625664 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:47.625833 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:47.626589 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:47.626627 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:47.627918 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:47.632260 (+ 4342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:47.632263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:47.632266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:47.632704 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:47.633192 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:47.633200 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:47.633207 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:47.633409 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:47.634113 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:47.634119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:47.634308 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 02:52:47.634470 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 02:52:47.634818 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"thread_start_us":889,"threads_started":1}
I20260926 02:52:47.645562 22172 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:37997
I20260926 02:52:47.645602 22306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:37997 every 8 connection(s)
I20260926 02:52:47.646787 22172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:52:47.648955 22307 heartbeater.cc:347] Connected to a master server at 127.19.185.254:42001
I20260926 02:52:47.649231 22307 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:47.649902 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 22172
I20260926 02:52:47.650043 22307 heartbeater.cc:511] Master 127.19.185.254:42001 requested a full tablet report, sending...
I20260926 02:52:47.650086 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:52:47.650669 20199 external_mini_cluster.cc:1428] Setting key 6286d1bf63e6821322b451595e64803e
I20260926 02:52:47.651674 21961 ts_manager.cc:194] Registered new tserver with Master: f265e515f6744cde815a3ca1b4932376 (127.19.185.194:37997)
I20260926 02:52:47.652869 21961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:45987
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:52:47.694518 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:42001
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:48.030635 22311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:48.030906 22311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:48.031055 22311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:48.038615 22311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:48.038756 22311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:48.038872 22311 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:52:48.038946 22311 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:52:48.039086 22311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:52:48.053999 22311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.185.254:42001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:48.055768 22311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:48.057651 22311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:48.075055 22319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:48.075287 22316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:48.075613 22317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:48.077339 22311 server_base.cc:1034] running on GCE node
I20260926 02:52:48.077965 22311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:48.078915 22311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:48.080144 22311 hybrid_clock.cc:648] HybridClock initialized: now 1790391168080072 us; error 64 us; skew 500 ppm
I20260926 02:52:48.080662 22311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:48 dist-test-slave-ktng krb5kdc[21916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391168, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:48.085260 22311 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:52:48.087210 22311 webserver.cc:460] Webserver started at http://127.19.185.195:38481/ using document root <none> and password file <none>
I20260926 02:52:48.088292 22311 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:48.088449 22311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:48.088851 22311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:48.092763 22311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "7114a2baac384b11a4e89008cbc3e1eb"
format_stamp: "Formatted at 2026-09-26 02:52:48 on dist-test-slave-ktng"
server_key: "4a3c687b8a8c5b8670378c63ee7579fd"
server_key_iv: "788c12d2574ab3cb37b309fbaef72f59"
server_key_version: "encryptionkey@0"
I20260926 02:52:48.093485 22311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "7114a2baac384b11a4e89008cbc3e1eb"
format_stamp: "Formatted at 2026-09-26 02:52:48 on dist-test-slave-ktng"
server_key: "4a3c687b8a8c5b8670378c63ee7579fd"
server_key_iv: "788c12d2574ab3cb37b309fbaef72f59"
server_key_version: "encryptionkey@0"
I20260926 02:52:48.098802 22311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:52:48.102283 22326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:48.103631 22311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:52:48.103799 22311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "7114a2baac384b11a4e89008cbc3e1eb"
format_stamp: "Formatted at 2026-09-26 02:52:48 on dist-test-slave-ktng"
server_key: "4a3c687b8a8c5b8670378c63ee7579fd"
server_key_iv: "788c12d2574ab3cb37b309fbaef72f59"
server_key_version: "encryptionkey@0"
I20260926 02:52:48.104017 22311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:48.126327 22311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:48.127995 22168 heartbeater.cc:503] Master 127.19.185.254:42001 was elected leader, sending a full tablet report...
I20260926 02:52:48.130906 22311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:48.131134 22311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:48.133859 22311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:48.133988 22311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:48.134119 22311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:48.134227 22311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:52:48 dist-test-slave-ktng krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391168, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:48.162982 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.134645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:57147 (local address 127.19.185.254:42001)
0926 02:52:48.134842 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:48.134848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:48.136480 (+ 1632us) server_negotiation.cc:407] Connection header received
0926 02:52:48.139719 (+ 3239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:48.139727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:48.139831 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:48.139927 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:48.142852 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.143954 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.146755 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.147164 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.147301 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.152958 (+ 5657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:48.152967 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:48.152975 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:48.153021 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:48.157105 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:48.158095 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:48.158104 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:48.158110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:48.158307 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:48.158755 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:48.158762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:48.158768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:48.159175 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:48.159387 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:48.162220 (+ 2833us) server_negotiation.cc:299] Negotiation successful
0926 02:52:48.162548 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:52:48.163035 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.135268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:42001 (local address 127.19.185.195:57147)
0926 02:52:48.136268 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:48.136341 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:48.139316 (+ 2975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:48.140565 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:48.140584 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:48.141539 (+ 955us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:48.142634 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.142678 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.144166 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.144172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.146567 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.146579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.147445 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.147455 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.147690 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.148931 (+ 1241us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:48.148956 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:48.152701 (+ 3745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:48.157263 (+ 4562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:48.157276 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:48.157302 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:48.157933 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:48.158431 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:48.158434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:48.158437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:48.158659 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:48.159373 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:48.159386 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:48.160008 (+ 622us) client_negotiation.cc:769] Sending connection context
0926 02:52:48.160422 (+ 414us) client_negotiation.cc:240] Negotiation successful
0926 02:52:48.161130 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":382,"threads_started":1}
I20260926 02:52:48.177891 22311 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:40653
I20260926 02:52:48.177996 22445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:40653 every 8 connection(s)
I20260926 02:52:48.179672 22311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:52:48.182176 22446 heartbeater.cc:347] Connected to a master server at 127.19.185.254:42001
I20260926 02:52:48.182417 22446 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:48.182809 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 22311
I20260926 02:52:48.183003 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:52:48.183132 22446 heartbeater.cc:511] Master 127.19.185.254:42001 requested a full tablet report, sending...
I20260926 02:52:48.183640 20199 external_mini_cluster.cc:1428] Setting key 60164251a0a671ac5a1da649c45f53d7
I20260926 02:52:48.184789 21961 ts_manager.cc:194] Registered new tserver with Master: 7114a2baac384b11a4e89008cbc3e1eb (127.19.185.195:40653)
I20260926 02:52:48.185834 21961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:57147
I20260926 02:52:48.186856 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:52:48.187282 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.172019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:58015 (local address 127.19.185.254:42001)
0926 02:52:48.172251 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:48.172258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:48.172382 (+ 124us) server_negotiation.cc:407] Connection header received
0926 02:52:48.172553 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:48.172561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:48.172646 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:48.172760 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:48.174385 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.175699 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.177881 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.178446 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.178545 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.181007 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:48.181012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:48.181016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:48.181044 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:48.184411 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:48.185214 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:48.185221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:48.185226 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:48.185397 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:48.185781 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:48.185786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:48.185789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:48.186129 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:48.186279 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:48.186709 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 02:52:48.186924 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:52:48.187424 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.171515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:42001 (local address 127.19.185.195:58015)
0926 02:52:48.172261 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:48.172288 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:48.172431 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:48.172815 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:48.172823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:48.173683 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:48.174207 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.174223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.175894 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.175902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.177692 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.177713 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.178630 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.178636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.178813 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.179587 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:48.179612 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:48.180809 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:48.184547 (+ 3738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:48.184554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:48.184558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:48.185058 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:48.185527 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:48.185533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:48.185537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:48.185663 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:48.186237 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:48.186242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:48.186455 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 02:52:48.186606 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 02:52:48.186924 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":548,"threads_started":1}
Sep 26 02:52:48 dist-test-slave-ktng krb5kdc[21916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:52:48 dist-test-slave-ktng krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:48.234066 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.219936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.19.185.254:42001)
0926 02:52:48.220164 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:48.220171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:48.220449 (+ 278us) server_negotiation.cc:407] Connection header received
0926 02:52:48.220668 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:48.220674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:48.220751 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:48.220862 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:48.222251 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.223006 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.224333 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.224763 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.224902 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.228795 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:48.228800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:48.228803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:48.228834 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:48.231430 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:48.232061 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:48.232068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:48.232071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:48.232202 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:48.232574 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:48.232580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:48.232584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:48.232874 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:48.233118 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:48.233430 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 02:52:48.233667 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:52:48.238404 21960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:52:48.246039 21960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:52:48.283321 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.259608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34502 (local address 127.19.185.194:37997)
0926 02:52:48.275780 (+ 16172us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:48.275792 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:48.275838 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:52:48.275969 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:48.275981 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:48.276347 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:48.276592 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:48.277456 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.278733 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.281158 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.282013 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.282244 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.282432 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:48.282809 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 02:52:48.282997 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15965,"thread_start_us":15818,"threads_started":1}
I20260926 02:52:48.283387 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.259070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:37997 (local address 127.0.0.1:34502)
0926 02:52:48.272381 (+ 13311us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:48.272428 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:48.272615 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:48.276843 (+ 4228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:48.276851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:48.276879 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:48.277252 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.277267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.278859 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.278867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.280995 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.281007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.282253 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.282259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.282540 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.282601 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 02:52:48.282748 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 02:52:48.282945 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13057,"mutex_wait_us":38,"thread_start_us":12885,"threads_started":1}
I20260926 02:52:48.285259 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.259449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:36921 (local address 127.0.0.1:41616)
0926 02:52:48.276130 (+ 16681us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:48.276146 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:48.276261 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:48.277265 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:48.277270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:48.277297 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:48.277766 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.277781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.279951 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.279957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.282580 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.282589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.284534 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.284539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.284647 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.284670 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:52:48.284789 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 02:52:48.284910 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16563,"mutex_wait_us":75,"thread_start_us":16395,"threads_started":1}
I20260926 02:52:48.285288 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.260166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.19.185.193:36921)
0926 02:52:48.275235 (+ 15069us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:48.275248 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:48.276409 (+ 1161us) server_negotiation.cc:407] Connection header received
0926 02:52:48.276520 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:48.276528 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:48.276869 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:48.277081 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:48.277982 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.279768 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.282759 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.284309 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.284477 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.284649 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:48.284844 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 02:52:48.285014 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14907,"thread_start_us":14761,"threads_started":1}
I20260926 02:52:48.287529 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.259102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:40653 (local address 127.0.0.1:55256)
0926 02:52:48.274347 (+ 15245us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:48.274371 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:48.274511 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:48.277836 (+ 3325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:48.277841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:48.277866 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:48.278196 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.278206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.280170 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.280177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.282568 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.282584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.286704 (+ 4120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.286711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.286882 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.286911 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 02:52:48.286998 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 02:52:48.287130 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15055,"mutex_wait_us":82,"thread_start_us":15504,"threads_started":1}
I20260926 02:52:48.288235 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.259556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55256 (local address 127.19.185.195:40653)
0926 02:52:48.277264 (+ 17708us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:48.277272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:48.277300 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:52:48.277397 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:48.277404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:48.277623 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:48.277828 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:48.278412 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.279982 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.284744 (+ 4762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.286236 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.286420 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.286570 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:48.287747 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0926 02:52:48.287906 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17549,"thread_start_us":17424,"threads_started":1}
I20260926 02:52:48.287912 22237 tablet_service.cc:1467] Processing CreateTablet for tablet 4bafe45547b14ac49e1db7a2e85edb22 (DEFAULT_TABLE table=test-table [id=6bf2a6df1eca4d7b932bfad058f2f793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:48.288749 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 4bafe45547b14ac49e1db7a2e85edb22 (DEFAULT_TABLE table=test-table [id=6bf2a6df1eca4d7b932bfad058f2f793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:48.289418 22237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bafe45547b14ac49e1db7a2e85edb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:48.290061 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bafe45547b14ac49e1db7a2e85edb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:48.291778 22376 tablet_service.cc:1467] Processing CreateTablet for tablet 4bafe45547b14ac49e1db7a2e85edb22 (DEFAULT_TABLE table=test-table [id=6bf2a6df1eca4d7b932bfad058f2f793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:48.293090 22376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bafe45547b14ac49e1db7a2e85edb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:48.301323 22465 tablet_bootstrap.cc:492] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f: Bootstrap starting.
I20260926 02:52:48.303966 22466 tablet_bootstrap.cc:492] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376: Bootstrap starting.
I20260926 02:52:48.304868 22465 tablet_bootstrap.cc:654] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:48.306576 22465 log.cc:824] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:52:48.306846 22467 tablet_bootstrap.cc:492] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb: Bootstrap starting.
I20260926 02:52:48.307878 22466 tablet_bootstrap.cc:654] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:48.309088 22466 log.cc:824] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376: Log is configured to *not* fsync() on all Append() calls
I20260926 02:52:48.310089 22465 tablet_bootstrap.cc:492] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f: No bootstrap required, opened a new log
I20260926 02:52:48.310243 22467 tablet_bootstrap.cc:654] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:48.310390 22465 ts_tablet_manager.cc:1397] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:52:48.311224 22467 log.cc:824] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb: Log is configured to *not* fsync() on all Append() calls
I20260926 02:52:48.312175 22466 tablet_bootstrap.cc:492] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376: No bootstrap required, opened a new log
I20260926 02:52:48.312428 22466 ts_tablet_manager.cc:1397] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 02:52:48.313781 22467 tablet_bootstrap.cc:492] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb: No bootstrap required, opened a new log
I20260926 02:52:48.314064 22467 ts_tablet_manager.cc:1397] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.004s
I20260926 02:52:48.321317 22467 raft_consensus.cc:348] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } }
I20260926 02:52:48.321372 22465 raft_consensus.cc:348] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } }
I20260926 02:52:48.321683 22467 raft_consensus.cc:374] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:48.321810 22467 raft_consensus.cc:729] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7114a2baac384b11a4e89008cbc3e1eb, State: Initialized, Role: FOLLOWER
I20260926 02:52:48.321812 22465 raft_consensus.cc:374] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:48.321926 22465 raft_consensus.cc:729] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed52fbfc571b4439a2683dbbb31dcd9f, State: Initialized, Role: FOLLOWER
I20260926 02:52:48.322412 22467 consensus_queue.cc:260] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } }
I20260926 02:52:48.322562 22465 consensus_queue.cc:260] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } }
I20260926 02:52:48.323282 22466 raft_consensus.cc:348] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } }
I20260926 02:52:48.323761 22466 raft_consensus.cc:374] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:48.323869 22466 raft_consensus.cc:729] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f265e515f6744cde815a3ca1b4932376, State: Initialized, Role: FOLLOWER
I20260926 02:52:48.324445 22466 consensus_queue.cc:260] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } }
I20260926 02:52:48.326319 22465 ts_tablet_manager.cc:1428] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260926 02:52:48.327374 22307 heartbeater.cc:503] Master 127.19.185.254:42001 was elected leader, sending a full tablet report...
I20260926 02:52:48.328030 22466 ts_tablet_manager.cc:1428] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260926 02:52:48.329700 22446 heartbeater.cc:503] Master 127.19.185.254:42001 was elected leader, sending a full tablet report...
I20260926 02:52:48.330340 22467 ts_tablet_manager.cc:1428] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb: Time spent starting tablet: real 0.016s user 0.000s sys 0.015s
W20260926 02:52:48.371250 22169 tablet.cc:2369] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:52:48.398962 22308 tablet.cc:2369] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:52:48.424587 22472 raft_consensus.cc:482] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:52:48.424885 22472 raft_consensus.cc:504] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } }
I20260926 02:52:48.426616 22472 leader_election.cc:288] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f265e515f6744cde815a3ca1b4932376 (127.19.185.194:37997), 7114a2baac384b11a4e89008cbc3e1eb (127.19.185.195:40653)
W20260926 02:52:48.431475 22447 tablet.cc:2369] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:52:48.434329 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.427517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:57629 (local address 127.19.185.195:40653)
0926 02:52:48.427728 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:48.427734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:48.427905 (+ 171us) server_negotiation.cc:407] Connection header received
0926 02:52:48.428081 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:48.428088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:48.428165 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:48.428263 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:48.428864 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.429585 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.431964 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.433198 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.433372 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.433446 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:48.433942 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 02:52:48.434068 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:52:48.434329 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.427008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:38945 (local address 127.19.185.194:37997)
0926 02:52:48.427242 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:48.427248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:48.427628 (+ 380us) server_negotiation.cc:407] Connection header received
0926 02:52:48.427928 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:48.427933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:48.428013 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:48.428098 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:48.428753 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.429494 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.432066 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.433245 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.433411 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.433498 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:48.433942 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 02:52:48.434071 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:52:48.434504 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.426963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:40653 (local address 127.19.185.193:57629)
0926 02:52:48.427760 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:48.427778 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:48.427936 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:48.428296 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:48.428299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:48.428337 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:48.428710 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.428723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.429762 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.429767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.431813 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.431825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.433473 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.433485 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.433724 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.433824 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 02:52:48.433901 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 02:52:48.434052 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":129,"spinlock_wait_cycles":1792,"thread_start_us":603,"threads_started":1}
I20260926 02:52:48.434537 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.426806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:37997 (local address 127.19.185.193:38945)
0926 02:52:48.427489 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:48.427514 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:48.427761 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:48.428129 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:48.428133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:48.428157 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:48.428604 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.428615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.429624 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.429632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.431918 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.431932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.433498 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.433503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.433723 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.433797 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 02:52:48.433901 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 02:52:48.434093 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":136,"spinlock_wait_cycles":6784,"thread_start_us":356,"threads_started":1}
I20260926 02:52:48.435608 22262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bafe45547b14ac49e1db7a2e85edb22" candidate_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f265e515f6744cde815a3ca1b4932376" is_pre_election: true
I20260926 02:52:48.435706 22396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bafe45547b14ac49e1db7a2e85edb22" candidate_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7114a2baac384b11a4e89008cbc3e1eb" is_pre_election: true
I20260926 02:52:48.436095 22262 raft_consensus.cc:2454] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed52fbfc571b4439a2683dbbb31dcd9f in term 0.
I20260926 02:52:48.436228 22396 raft_consensus.cc:2454] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed52fbfc571b4439a2683dbbb31dcd9f in term 0.
I20260926 02:52:48.436807 22051 leader_election.cc:302] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7114a2baac384b11a4e89008cbc3e1eb, ed52fbfc571b4439a2683dbbb31dcd9f, f265e515f6744cde815a3ca1b4932376; no voters:
I20260926 02:52:48.437194 22472 raft_consensus.cc:2781] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:52:48.437341 22472 raft_consensus.cc:482] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:52:48.437436 22472 raft_consensus.cc:3037] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:48.439806 22472 raft_consensus.cc:504] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } }
I20260926 02:52:48.440414 22472 leader_election.cc:288] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [CANDIDATE]: Term 1 election: Requested vote from peers f265e515f6744cde815a3ca1b4932376 (127.19.185.194:37997), 7114a2baac384b11a4e89008cbc3e1eb (127.19.185.195:40653)
I20260926 02:52:48.440822 22262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bafe45547b14ac49e1db7a2e85edb22" candidate_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f265e515f6744cde815a3ca1b4932376"
I20260926 02:52:48.440888 22396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bafe45547b14ac49e1db7a2e85edb22" candidate_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7114a2baac384b11a4e89008cbc3e1eb"
I20260926 02:52:48.441031 22262 raft_consensus.cc:3037] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:48.441056 22396 raft_consensus.cc:3037] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:48.444262 22262 raft_consensus.cc:2454] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed52fbfc571b4439a2683dbbb31dcd9f in term 1.
I20260926 02:52:48.444262 22396 raft_consensus.cc:2454] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed52fbfc571b4439a2683dbbb31dcd9f in term 1.
I20260926 02:52:48.444650 22051 leader_election.cc:302] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ed52fbfc571b4439a2683dbbb31dcd9f, f265e515f6744cde815a3ca1b4932376; no voters:
I20260926 02:52:48.444936 22472 raft_consensus.cc:2781] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:48.445380 22472 raft_consensus.cc:686] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 LEADER]: Becoming Leader. State: Replica: ed52fbfc571b4439a2683dbbb31dcd9f, State: Running, Role: LEADER
I20260926 02:52:48.445900 22472 consensus_queue.cc:237] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } }
I20260926 02:52:48.450349 21960 catalog_manager.cc:5495] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f reported cstate change: term changed from 0 to 1, leader changed from <none> to ed52fbfc571b4439a2683dbbb31dcd9f (127.19.185.193). New cstate: current_term: 1 leader_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } health_report { overall_health: HEALTHY } } }
Sep 26 02:52:48 dist-test-slave-ktng krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391166, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:48.510411 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.494086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40632 (local address 127.19.185.254:42001)
0926 02:52:48.494292 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:48.494328 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:48.495066 (+ 738us) server_negotiation.cc:407] Connection header received
0926 02:52:48.495426 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:48.495431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:48.495575 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:48.495694 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:48.497210 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.498301 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.500227 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.500691 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.500784 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.504636 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:48.504641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:48.504644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:48.504671 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:48.507582 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:48.508372 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:48.508379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:48.508384 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:48.508504 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:48.508900 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:48.508905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:48.508909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:48.509217 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:48.509368 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:48.509780 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 02:52:48.510104 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":69}
I20260926 02:52:48.510443 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.494154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:42001 (local address 127.0.0.1:40632)
0926 02:52:48.494964 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:48.494987 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:48.495180 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:48.495750 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:48.495756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:48.496702 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:48.497041 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.497053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.498423 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.498431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.500079 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.500093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.500871 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.500877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.501057 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.501747 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:48.501770 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:48.504443 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:48.507736 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:48.507749 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:48.507757 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:48.508248 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:48.508633 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:48.508637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:48.508641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:48.508784 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:48.509348 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:48.509357 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:48.509523 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 02:52:48.509697 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 02:52:48.509978 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":482,"threads_started":1}
I20260926 02:52:48.515481 21960 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:40632:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 02:52:48.516462 21960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:52:48.522986 22237 tablet_service.cc:1467] Processing CreateTablet for tablet a0f841d6ab434643af19c116fd0f34c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=04c97ad75a064ba9836cff833055a601]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:52:48.522851 22098 tablet_service.cc:1467] Processing CreateTablet for tablet a0f841d6ab434643af19c116fd0f34c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=04c97ad75a064ba9836cff833055a601]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:52:48.523258 22376 tablet_service.cc:1467] Processing CreateTablet for tablet a0f841d6ab434643af19c116fd0f34c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=04c97ad75a064ba9836cff833055a601]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:52:48.523912 22237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f841d6ab434643af19c116fd0f34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:48.523954 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f841d6ab434643af19c116fd0f34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:48.524022 22376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f841d6ab434643af19c116fd0f34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:48.529753 22466 tablet_bootstrap.cc:492] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376: Bootstrap starting.
I20260926 02:52:48.531167 22466 tablet_bootstrap.cc:654] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:48.531193 22465 tablet_bootstrap.cc:492] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f: Bootstrap starting.
I20260926 02:52:48.531883 22467 tablet_bootstrap.cc:492] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb: Bootstrap starting.
I20260926 02:52:48.533027 22465 tablet_bootstrap.cc:654] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:48.533226 22467 tablet_bootstrap.cc:654] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:48.535825 22465 tablet_bootstrap.cc:492] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f: No bootstrap required, opened a new log
I20260926 02:52:48.535964 22465 ts_tablet_manager.cc:1397] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 02:52:48.536015 22466 tablet_bootstrap.cc:492] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376: No bootstrap required, opened a new log
I20260926 02:52:48.536023 22467 tablet_bootstrap.cc:492] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb: No bootstrap required, opened a new log
I20260926 02:52:48.536134 22466 ts_tablet_manager.cc:1397] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260926 02:52:48.536152 22467 ts_tablet_manager.cc:1397] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260926 02:52:48.536757 22465 raft_consensus.cc:348] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:48.536823 22466 raft_consensus.cc:348] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:48.537002 22465 raft_consensus.cc:374] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:48.537056 22466 raft_consensus.cc:374] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:48.536897 22467 raft_consensus.cc:348] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:48.537103 22465 raft_consensus.cc:729] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed52fbfc571b4439a2683dbbb31dcd9f, State: Initialized, Role: FOLLOWER
I20260926 02:52:48.537153 22467 raft_consensus.cc:374] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:48.537156 22466 raft_consensus.cc:729] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f265e515f6744cde815a3ca1b4932376, State: Initialized, Role: FOLLOWER
I20260926 02:52:48.537276 22467 raft_consensus.cc:729] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7114a2baac384b11a4e89008cbc3e1eb, State: Initialized, Role: FOLLOWER
I20260926 02:52:48.537357 22465 consensus_queue.cc:260] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:48.537441 22466 consensus_queue.cc:260] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:48.537537 22467 consensus_queue.cc:260] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:48.537998 22466 ts_tablet_manager.cc:1428] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:52:48.538003 22465 ts_tablet_manager.cc:1428] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 02:52:48.538122 22467 ts_tablet_manager.cc:1428] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:52:48.538543 22473 tablet_replica.cc:439] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } }
I20260926 02:52:48.538729 22472 tablet_replica.cc:439] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } }
I20260926 02:52:48.538932 22473 tablet_replica.cc:442] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:52:48.538981 22472 tablet_replica.cc:442] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:52:48.539234 22471 tablet_replica.cc:439] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } }
I20260926 02:52:48.539572 22471 tablet_replica.cc:442] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:52:48.788534 22472 raft_consensus.cc:482] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:52:48.788802 22472 raft_consensus.cc:504] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:48.789527 22472 leader_election.cc:288] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f265e515f6744cde815a3ca1b4932376 (127.19.185.194:37997), 7114a2baac384b11a4e89008cbc3e1eb (127.19.185.195:40653)
I20260926 02:52:48.790143 22262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f841d6ab434643af19c116fd0f34c6" candidate_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f265e515f6744cde815a3ca1b4932376" is_pre_election: true
I20260926 02:52:48.790405 22262 raft_consensus.cc:2454] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed52fbfc571b4439a2683dbbb31dcd9f in term 0.
I20260926 02:52:48.790350 22396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f841d6ab434643af19c116fd0f34c6" candidate_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7114a2baac384b11a4e89008cbc3e1eb" is_pre_election: true
I20260926 02:52:48.790596 22396 raft_consensus.cc:2454] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed52fbfc571b4439a2683dbbb31dcd9f in term 0.
I20260926 02:52:48.790877 22051 leader_election.cc:302] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ed52fbfc571b4439a2683dbbb31dcd9f, f265e515f6744cde815a3ca1b4932376; no voters:
I20260926 02:52:48.791179 22472 raft_consensus.cc:2781] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:52:48.791263 22472 raft_consensus.cc:482] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:52:48.791428 22472 raft_consensus.cc:3037] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:48.793498 22472 raft_consensus.cc:504] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:48.794122 22472 leader_election.cc:288] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [CANDIDATE]: Term 1 election: Requested vote from peers f265e515f6744cde815a3ca1b4932376 (127.19.185.194:37997), 7114a2baac384b11a4e89008cbc3e1eb (127.19.185.195:40653)
I20260926 02:52:48.794497 22262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f841d6ab434643af19c116fd0f34c6" candidate_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f265e515f6744cde815a3ca1b4932376"
I20260926 02:52:48.794502 22396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f841d6ab434643af19c116fd0f34c6" candidate_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7114a2baac384b11a4e89008cbc3e1eb"
I20260926 02:52:48.794713 22262 raft_consensus.cc:3037] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:48.794737 22396 raft_consensus.cc:3037] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:48.796787 22262 raft_consensus.cc:2454] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed52fbfc571b4439a2683dbbb31dcd9f in term 1.
I20260926 02:52:48.796831 22396 raft_consensus.cc:2454] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed52fbfc571b4439a2683dbbb31dcd9f in term 1.
I20260926 02:52:48.797163 22051 leader_election.cc:302] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ed52fbfc571b4439a2683dbbb31dcd9f, f265e515f6744cde815a3ca1b4932376; no voters:
I20260926 02:52:48.797420 22472 raft_consensus.cc:2781] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:48.797626 22472 raft_consensus.cc:686] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 LEADER]: Becoming Leader. State: Replica: ed52fbfc571b4439a2683dbbb31dcd9f, State: Running, Role: LEADER
I20260926 02:52:48.797905 22472 consensus_queue.cc:237] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:48.799556 22476 tablet_replica.cc:439] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f: TxnStatusTablet state changed. Reason: New leader ed52fbfc571b4439a2683dbbb31dcd9f. Latest consensus state: current_term: 1 leader_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } }
I20260926 02:52:48.799851 22476 tablet_replica.cc:442] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:52:48.800638 22488 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 02:52:48.800915 22488 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 02:52:48.801301 21961 catalog_manager.cc:5495] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f reported cstate change: term changed from 0 to 1, leader changed from <none> to ed52fbfc571b4439a2683dbbb31dcd9f (127.19.185.193). New cstate: current_term: 1 leader_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } health_report { overall_health: UNKNOWN } } }
I20260926 02:52:48.833729 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.824937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:36921 (local address 127.0.0.1:41624)
0926 02:52:48.825193 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:48.825214 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:48.825382 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:48.828209 (+ 2827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:48.828212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:48.828222 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:52:48.828368 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.828375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.829943 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.829947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.830924 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.830936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.831908 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.831912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.832043 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.832098 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:52:48.833265 (+ 1167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:52:48.833279 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 02:52:48.833329 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 02:52:48.833496 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 02:52:48.833849 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.825051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.19.185.193:36921)
0926 02:52:48.827744 (+ 2693us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:48.827750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:48.827770 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:52:48.827873 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:48.827880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:48.827997 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:48.828175 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:52:48.828531 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.829818 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.831075 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.831758 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.831886 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.832328 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:52:48.833160 (+ 832us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:52:48.833315 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:48.833374 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 02:52:48.833455 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2558,"thread_start_us":2407,"threads_started":1}
I20260926 02:52:48.842617 22396 raft_consensus.cc:1261] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Refusing update from remote peer ed52fbfc571b4439a2683dbbb31dcd9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:48.842599 22262 raft_consensus.cc:1261] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Refusing update from remote peer ed52fbfc571b4439a2683dbbb31dcd9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:48.843673 22472 consensus_queue.cc:1035] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:48.843974 22476 consensus_queue.cc:1035] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:48.852504 22471 tablet_replica.cc:439] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb: TxnStatusTablet state changed. Reason: New leader ed52fbfc571b4439a2683dbbb31dcd9f. Latest consensus state: current_term: 1 leader_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } }
I20260926 02:52:48.852769 22471 tablet_replica.cc:442] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:52:48.856796 22492 mvcc.cc:205] Tried to move back new op lower bound from 7333442227569442816 to 7333442227398238208. Current Snapshot: MvccSnapshot[applied={T|T < 7333442227569442816}]
I20260926 02:52:48.856685 22473 tablet_replica.cc:439] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376: TxnStatusTablet state changed. Reason: New leader ed52fbfc571b4439a2683dbbb31dcd9f. Latest consensus state: current_term: 1 leader_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } }
I20260926 02:52:48.856985 22473 tablet_replica.cc:442] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:52:48.858565 22493 mvcc.cc:205] Tried to move back new op lower bound from 7333442227569442816 to 7333442227398238208. Current Snapshot: MvccSnapshot[applied={T|T < 7333442227569442816}]
I20260926 02:52:48.861809 22494 mvcc.cc:205] Tried to move back new op lower bound from 7333442227569442816 to 7333442227398238208. Current Snapshot: MvccSnapshot[applied={T|T < 7333442227569442816}]
I20260926 02:52:48.866729 22473 tablet_replica.cc:439] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } }
I20260926 02:52:48.866961 22473 tablet_replica.cc:442] T a0f841d6ab434643af19c116fd0f34c6 P f265e515f6744cde815a3ca1b4932376: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:52:48.868818 22471 tablet_replica.cc:439] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } }
I20260926 02:52:48.869160 22471 tablet_replica.cc:442] T a0f841d6ab434643af19c116fd0f34c6 P 7114a2baac384b11a4e89008cbc3e1eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:52:48.875927 22472 tablet_replica.cc:439] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } }
I20260926 02:52:48.876309 22472 tablet_replica.cc:442] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:52:48.876700 22500 tablet_replica.cc:439] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } } }
I20260926 02:52:48.877187 22500 tablet_replica.cc:442] T a0f841d6ab434643af19c116fd0f34c6 P ed52fbfc571b4439a2683dbbb31dcd9f: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:52:48.886123 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.880376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.19.185.193:36921)
0926 02:52:48.880550 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:48.880557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:48.880823 (+ 266us) server_negotiation.cc:407] Connection header received
0926 02:52:48.881005 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:48.881010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:48.881115 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:48.881213 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:52:48.881580 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.882747 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.884142 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.884841 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.884960 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.885344 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:52:48.885483 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:52:48.885582 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:48.885711 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 02:52:48.885794 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 02:52:48.904943 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.898318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:43993 (local address 127.19.185.193:36921)
0926 02:52:48.898558 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:48.898568 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:48.899177 (+ 609us) server_negotiation.cc:407] Connection header received
0926 02:52:48.899416 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:48.899420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:48.899483 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:48.899563 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:52:48.900190 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.901069 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.903164 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.903654 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.903749 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.904309 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:52:48.904401 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:52:48.904464 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:48.904614 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 02:52:48.904674 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:52:48.905279 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:48.898300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:36921 (local address 127.19.185.193:43993)
0926 02:52:48.899052 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:48.899073 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:48.899249 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:48.899601 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:48.899606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:48.899625 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:52:48.899980 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.899992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.901199 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.901205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.903020 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:48.903036 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:48.903835 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:48.903842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:48.904015 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:48.904091 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:52:48.904496 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:52:48.904507 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 02:52:48.904592 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 02:52:48.904740 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":381,"threads_started":1}
I20260926 02:52:48.914811 22262 raft_consensus.cc:1261] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Refusing update from remote peer ed52fbfc571b4439a2683dbbb31dcd9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:48.914819 22396 raft_consensus.cc:1261] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Refusing update from remote peer ed52fbfc571b4439a2683dbbb31dcd9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:48.915477 22500 consensus_queue.cc:1035] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:48.915819 22476 consensus_queue.cc:1035] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:48.978356 21960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:52:48.978657 21960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:52:48.982393 21960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76: Sending DeleteTablet for 3 replicas of tablet 4bafe45547b14ac49e1db7a2e85edb22
I20260926 02:52:48.983686 22237 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bafe45547b14ac49e1db7a2e85edb22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:48 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:34502
I20260926 02:52:48.983711 22376 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bafe45547b14ac49e1db7a2e85edb22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:48 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:55256
I20260926 02:52:48.984149 22098 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bafe45547b14ac49e1db7a2e85edb22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:48 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:41616
I20260926 02:52:48.986506 21960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:52:48.987540 21960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:52:48.996292 22511 tablet_replica.cc:330] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f: stopping tablet replica
I20260926 02:52:48.996325 22510 tablet_replica.cc:330] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376: stopping tablet replica
I20260926 02:52:48.996809 22510 raft_consensus.cc:2229] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:52:48.996816 22511 raft_consensus.cc:2229] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:52:48.998380 22375 tablet_service.cc:1467] Processing CreateTablet for tablet 4710ad3a56cc4fc3a25f0e3ea42849d2 (DEFAULT_TABLE table=test-table [id=53a9037167564f9fb0ae5a145ab9bb9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:48.998984 22375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4710ad3a56cc4fc3a25f0e3ea42849d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:48.998301 22510 raft_consensus.cc:2258] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:49.003125 22509 tablet_replica.cc:330] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb: stopping tablet replica
I20260926 02:52:49.003479 22237 tablet_service.cc:1467] Processing CreateTablet for tablet 4710ad3a56cc4fc3a25f0e3ea42849d2 (DEFAULT_TABLE table=test-table [id=53a9037167564f9fb0ae5a145ab9bb9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:49.003655 22510 ts_tablet_manager.cc:1905] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:49.003786 22509 raft_consensus.cc:2229] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:52:49.004221 22509 raft_consensus.cc:2258] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:49.006187 22509 ts_tablet_manager.cc:1905] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:49.006899 22237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4710ad3a56cc4fc3a25f0e3ea42849d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:49.013665 22509 ts_tablet_manager.cc:1918] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:52:49.013901 22509 log.cc:1196] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/4bafe45547b14ac49e1db7a2e85edb22
I20260926 02:52:49.013908 22467 tablet_bootstrap.cc:492] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb: Bootstrap starting.
I20260926 02:52:49.014817 22509 ts_tablet_manager.cc:1939] T 4bafe45547b14ac49e1db7a2e85edb22 P 7114a2baac384b11a4e89008cbc3e1eb: Deleting consensus metadata
I20260926 02:52:49.015775 22467 tablet_bootstrap.cc:654] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:49.016808 22510 ts_tablet_manager.cc:1918] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:52:49.017006 22510 log.cc:1196] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/4bafe45547b14ac49e1db7a2e85edb22
I20260926 02:52:49.017642 22510 ts_tablet_manager.cc:1939] T 4bafe45547b14ac49e1db7a2e85edb22 P f265e515f6744cde815a3ca1b4932376: Deleting consensus metadata
I20260926 02:52:49.017665 21946 catalog_manager.cc:4841] TS 7114a2baac384b11a4e89008cbc3e1eb (127.19.185.195:40653): tablet 4bafe45547b14ac49e1db7a2e85edb22 (table test-table [id=6bf2a6df1eca4d7b932bfad058f2f793]) successfully deleted
I20260926 02:52:49.018545 22467 tablet_bootstrap.cc:492] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb: No bootstrap required, opened a new log
I20260926 02:52:49.018718 22467 ts_tablet_manager.cc:1397] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 02:52:49.019415 21948 catalog_manager.cc:4841] TS f265e515f6744cde815a3ca1b4932376 (127.19.185.194:37997): tablet 4bafe45547b14ac49e1db7a2e85edb22 (table test-table [id=6bf2a6df1eca4d7b932bfad058f2f793]) successfully deleted
I20260926 02:52:49.019491 22467 raft_consensus.cc:348] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:49.019745 22467 raft_consensus.cc:374] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:49.019840 22467 raft_consensus.cc:729] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7114a2baac384b11a4e89008cbc3e1eb, State: Initialized, Role: FOLLOWER
I20260926 02:52:48.998261 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 4710ad3a56cc4fc3a25f0e3ea42849d2 (DEFAULT_TABLE table=test-table [id=53a9037167564f9fb0ae5a145ab9bb9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:49.020068 22467 consensus_queue.cc:260] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:49.020378 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4710ad3a56cc4fc3a25f0e3ea42849d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:49.020738 22467 ts_tablet_manager.cc:1428] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:52:48.997635 22511 raft_consensus.cc:2258] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:49.024545 22511 ts_tablet_manager.cc:1905] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:49.027397 22465 tablet_bootstrap.cc:492] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f: Bootstrap starting.
I20260926 02:52:49.029181 22465 tablet_bootstrap.cc:654] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:49.031435 22466 tablet_bootstrap.cc:492] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376: Bootstrap starting.
I20260926 02:52:49.031759 22465 tablet_bootstrap.cc:492] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f: No bootstrap required, opened a new log
I20260926 02:52:49.031908 22465 ts_tablet_manager.cc:1397] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260926 02:52:49.032778 22511 ts_tablet_manager.cc:1918] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:52:49.032716 22465 raft_consensus.cc:348] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:49.032936 22511 log.cc:1196] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/4bafe45547b14ac49e1db7a2e85edb22
I20260926 02:52:49.032966 22465 raft_consensus.cc:374] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:49.033068 22465 raft_consensus.cc:729] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed52fbfc571b4439a2683dbbb31dcd9f, State: Initialized, Role: FOLLOWER
I20260926 02:52:49.033138 22466 tablet_bootstrap.cc:654] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:49.033416 22465 consensus_queue.cc:260] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:49.034314 22465 ts_tablet_manager.cc:1428] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:52:49.033594 22511 ts_tablet_manager.cc:1939] T 4bafe45547b14ac49e1db7a2e85edb22 P ed52fbfc571b4439a2683dbbb31dcd9f: Deleting consensus metadata
I20260926 02:52:49.036609 21945 catalog_manager.cc:4841] TS ed52fbfc571b4439a2683dbbb31dcd9f (127.19.185.193:36921): tablet 4bafe45547b14ac49e1db7a2e85edb22 (table test-table [id=6bf2a6df1eca4d7b932bfad058f2f793]) successfully deleted
I20260926 02:52:49.038233 22466 tablet_bootstrap.cc:492] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376: No bootstrap required, opened a new log
I20260926 02:52:49.038501 22466 ts_tablet_manager.cc:1397] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260926 02:52:49.039119 22466 raft_consensus.cc:348] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:49.039381 22466 raft_consensus.cc:374] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:49.039464 22466 raft_consensus.cc:729] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f265e515f6744cde815a3ca1b4932376, State: Initialized, Role: FOLLOWER
I20260926 02:52:49.039742 22466 consensus_queue.cc:260] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:49.040270 22466 ts_tablet_manager.cc:1428] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 02:52:49.179795 22471 raft_consensus.cc:482] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:52:49.180047 22471 raft_consensus.cc:504] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:49.181775 22471 leader_election.cc:288] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed52fbfc571b4439a2683dbbb31dcd9f (127.19.185.193:36921), f265e515f6744cde815a3ca1b4932376 (127.19.185.194:37997)
I20260926 02:52:49.203665 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:49.182381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:43841 (local address 127.19.185.193:36921)
0926 02:52:49.182589 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:49.182595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:49.193268 (+ 10673us) server_negotiation.cc:407] Connection header received
0926 02:52:49.194406 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:49.194415 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:49.194537 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:49.194623 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:49.196476 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.198000 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.200981 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.202397 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.202542 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:49.202618 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:49.203190 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 02:52:49.203361 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 02:52:49.204715 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:49.182000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:36921 (local address 127.19.185.195:43841)
0926 02:52:49.193100 (+ 11100us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:49.193130 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:49.194001 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:49.194896 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:49.194903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:49.194935 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:49.195975 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:49.195992 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.198262 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.198268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:49.200786 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:49.200800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.202795 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.202801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:49.202990 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:49.203024 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:52:49.204169 (+ 1145us) client_negotiation.cc:240] Negotiation successful
0926 02:52:49.204311 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10921,"thread_start_us":17304,"threads_started":1}
I20260926 02:52:49.205823 22123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4710ad3a56cc4fc3a25f0e3ea42849d2" candidate_uuid: "7114a2baac384b11a4e89008cbc3e1eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" is_pre_election: true
I20260926 02:52:49.206379 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:49.182199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:37997 (local address 127.19.185.195:40257)
0926 02:52:49.198539 (+ 16340us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:49.198560 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:49.198675 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:49.199727 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:49.199734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:49.199765 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:49.200196 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:49.200206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.201698 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.201705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:49.203808 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:49.203822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.205620 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.205624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:49.205735 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:49.205756 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 02:52:49.205870 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 02:52:49.206002 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16169,"thread_start_us":10577,"threads_started":1}
I20260926 02:52:49.206291 22123 raft_consensus.cc:2454] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7114a2baac384b11a4e89008cbc3e1eb in term 0.
I20260926 02:52:49.206650 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:49.182597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:40257 (local address 127.19.185.194:37997)
0926 02:52:49.199168 (+ 16571us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:49.199174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:49.199200 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:52:49.199270 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:49.199275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:49.199379 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:49.199481 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:49.200424 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.201542 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.204010 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.205248 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.205403 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:49.205476 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:49.205908 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 02:52:49.206036 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16433,"thread_start_us":16980,"threads_started":1}
I20260926 02:52:49.207041 22327 leader_election.cc:302] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7114a2baac384b11a4e89008cbc3e1eb, ed52fbfc571b4439a2683dbbb31dcd9f; no voters:
I20260926 02:52:49.207526 22471 raft_consensus.cc:2781] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:52:49.207470 22262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4710ad3a56cc4fc3a25f0e3ea42849d2" candidate_uuid: "7114a2baac384b11a4e89008cbc3e1eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f265e515f6744cde815a3ca1b4932376" is_pre_election: true
I20260926 02:52:49.207653 22471 raft_consensus.cc:482] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:52:49.207752 22471 raft_consensus.cc:3037] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:49.207705 22262 raft_consensus.cc:2454] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7114a2baac384b11a4e89008cbc3e1eb in term 0.
I20260926 02:52:49.210397 22471 raft_consensus.cc:504] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:49.211388 22471 leader_election.cc:288] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [CANDIDATE]: Term 1 election: Requested vote from peers ed52fbfc571b4439a2683dbbb31dcd9f (127.19.185.193:36921), f265e515f6744cde815a3ca1b4932376 (127.19.185.194:37997)
I20260926 02:52:49.211467 22123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4710ad3a56cc4fc3a25f0e3ea42849d2" candidate_uuid: "7114a2baac384b11a4e89008cbc3e1eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f"
I20260926 02:52:49.211536 22262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4710ad3a56cc4fc3a25f0e3ea42849d2" candidate_uuid: "7114a2baac384b11a4e89008cbc3e1eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f265e515f6744cde815a3ca1b4932376"
I20260926 02:52:49.211700 22123 raft_consensus.cc:3037] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:49.211742 22262 raft_consensus.cc:3037] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:49.214708 22262 raft_consensus.cc:2454] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7114a2baac384b11a4e89008cbc3e1eb in term 1.
I20260926 02:52:49.215142 22330 leader_election.cc:302] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7114a2baac384b11a4e89008cbc3e1eb, f265e515f6744cde815a3ca1b4932376; no voters:
I20260926 02:52:49.215164 22123 raft_consensus.cc:2454] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7114a2baac384b11a4e89008cbc3e1eb in term 1.
I20260926 02:52:49.215441 22471 raft_consensus.cc:2781] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:49.215991 22471 raft_consensus.cc:686] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 LEADER]: Becoming Leader. State: Replica: 7114a2baac384b11a4e89008cbc3e1eb, State: Running, Role: LEADER
I20260926 02:52:49.216419 22471 consensus_queue.cc:237] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:49.220257 21960 catalog_manager.cc:5495] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 7114a2baac384b11a4e89008cbc3e1eb (127.19.185.195). New cstate: current_term: 1 leader_uuid: "7114a2baac384b11a4e89008cbc3e1eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } health_report { overall_health: HEALTHY } } }
I20260926 02:52:49.271523 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:49.258459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.19.185.195:40653)
0926 02:52:49.266245 (+ 7786us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:49.266252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:49.266278 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:52:49.266376 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:49.266380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:49.266459 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:49.266645 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:52:49.267111 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.267956 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.269249 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.269667 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.269804 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:49.270260 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:52:49.270855 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:52:49.271036 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:49.271088 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 02:52:49.271163 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7626,"thread_start_us":7424,"threads_started":1}
I20260926 02:52:49.287770 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:49.282198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:45017 (local address 127.19.185.193:36921)
0926 02:52:49.282392 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:49.282404 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:49.282904 (+ 500us) server_negotiation.cc:407] Connection header received
0926 02:52:49.283134 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:49.283138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:49.283204 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:49.283304 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:52:49.283846 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.284594 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.285942 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.286400 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.286567 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:49.286962 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:52:49.287073 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:52:49.287170 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:49.287347 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 02:52:49.287430 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:52:49.287833 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:49.282143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:36921 (local address 127.19.185.195:45017)
0926 02:52:49.282790 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:49.282805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:49.282986 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:49.283356 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:49.283361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:49.283372 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:52:49.283685 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:49.283695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.284731 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.284740 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:49.285798 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:49.285809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.286582 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.286588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:49.286734 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:49.286784 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:52:49.287173 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:52:49.287186 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 02:52:49.287283 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 02:52:49.287432 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":375,"threads_started":1}
I20260926 02:52:49.296869 22262 raft_consensus.cc:1261] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Refusing update from remote peer 7114a2baac384b11a4e89008cbc3e1eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:49.296886 22123 raft_consensus.cc:1261] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Refusing update from remote peer 7114a2baac384b11a4e89008cbc3e1eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:49.297609 22518 consensus_queue.cc:1035] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:49.297979 22471 consensus_queue.cc:1035] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:49.333112 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:49.324902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:52427 (local address 127.19.185.195:40653)
0926 02:52:49.325061 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:49.325068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:49.325086 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:52:49.325148 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:49.325153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:49.325232 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:49.325320 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:52:49.326006 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.327126 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.329891 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.330804 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.330944 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:49.332321 (+ 1377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:52:49.332449 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:52:49.332559 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:49.332738 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 02:52:49.332820 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:52:49.333348 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:49.320981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:40653 (local address 127.19.185.193:52427)
0926 02:52:49.321361 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:49.321382 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:49.321505 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:49.325519 (+ 4014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:49.325524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:49.325538 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:52:49.325720 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:49.325731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.327339 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.327348 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:49.329667 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:49.329687 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:49.331792 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:49.331802 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:49.332057 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:49.332130 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:52:49.332583 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:52:49.332598 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 02:52:49.332699 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 02:52:49.332921 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
W20260926 02:52:49.334525 22493 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 02:52:50.927713 21961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:52:50.928014 21961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:52:50.932255 21961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76: Sending DeleteTablet for 3 replicas of tablet 4710ad3a56cc4fc3a25f0e3ea42849d2
I20260926 02:52:50.933065 22098 tablet_service.cc:1514] Processing DeleteTablet for tablet 4710ad3a56cc4fc3a25f0e3ea42849d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:50 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:41616
I20260926 02:52:50.933212 22237 tablet_service.cc:1514] Processing DeleteTablet for tablet 4710ad3a56cc4fc3a25f0e3ea42849d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:50 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:34502
I20260926 02:52:50.933280 22375 tablet_service.cc:1514] Processing DeleteTablet for tablet 4710ad3a56cc4fc3a25f0e3ea42849d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:50 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:55256
I20260926 02:52:50.944696 22554 tablet_replica.cc:330] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb: stopping tablet replica
I20260926 02:52:50.945170 22554 raft_consensus.cc:2229] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:52:50.946033 22554 raft_consensus.cc:2258] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:50.946668 22552 tablet_replica.cc:330] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f: stopping tablet replica
I20260926 02:52:50.947060 22553 tablet_replica.cc:330] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376: stopping tablet replica
I20260926 02:52:50.947115 22552 raft_consensus.cc:2229] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:52:50.947341 22553 raft_consensus.cc:2229] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:52:50.947444 22552 raft_consensus.cc:2258] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:50.947641 22553 raft_consensus.cc:2258] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:50.948518 22554 ts_tablet_manager.cc:1905] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:50.948813 22552 ts_tablet_manager.cc:1905] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:50.949697 22553 ts_tablet_manager.cc:1905] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 02:52:50 dist-test-slave-ktng krb5kdc[21916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391170, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260926 02:52:51.009696 22552 ts_tablet_manager.cc:1918] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:52:51.009696 22553 ts_tablet_manager.cc:1918] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:52:51.009857 22553 log.cc:1196] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/4710ad3a56cc4fc3a25f0e3ea42849d2
I20260926 02:52:51.009917 22552 log.cc:1196] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/4710ad3a56cc4fc3a25f0e3ea42849d2
I20260926 02:52:51.010329 22553 ts_tablet_manager.cc:1939] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P f265e515f6744cde815a3ca1b4932376: Deleting consensus metadata
I20260926 02:52:51.010371 22552 ts_tablet_manager.cc:1939] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P ed52fbfc571b4439a2683dbbb31dcd9f: Deleting consensus metadata
Sep 26 02:52:51 dist-test-slave-ktng krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391170, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:51.036864 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:51.012979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.19.185.254:42001)
0926 02:52:51.013551 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:51.013555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:51.013721 (+ 166us) server_negotiation.cc:407] Connection header received
0926 02:52:51.013882 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:51.013886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:51.013956 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:51.014057 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:51.015684 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.016482 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.017702 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.018132 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.018270 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:51.030898 (+ 12628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:51.030904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:51.030907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:51.030949 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:51.033802 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:51.034829 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:51.034834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:51.034836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:51.034982 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:51.035540 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:51.035543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:51.035546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:51.035825 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:51.036004 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:51.036391 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 02:52:51.036575 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":299,"threads_started":1}
I20260926 02:52:51.041165 21961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:52:51.041774 21961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:52:51.048905 22237 tablet_service.cc:1467] Processing CreateTablet for tablet 550ace0c7fea437b96d632a87000e5cb (DEFAULT_TABLE table=test-table [id=66883228173f49d48a216dff74cb1499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:51.048905 22098 tablet_service.cc:1467] Processing CreateTablet for tablet 550ace0c7fea437b96d632a87000e5cb (DEFAULT_TABLE table=test-table [id=66883228173f49d48a216dff74cb1499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:51.049362 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 550ace0c7fea437b96d632a87000e5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:51.049371 22237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 550ace0c7fea437b96d632a87000e5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:51.051445 22375 tablet_service.cc:1467] Processing CreateTablet for tablet 550ace0c7fea437b96d632a87000e5cb (DEFAULT_TABLE table=test-table [id=66883228173f49d48a216dff74cb1499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:51.056668 22563 tablet_bootstrap.cc:492] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376: Bootstrap starting.
I20260926 02:52:51.058332 22563 tablet_bootstrap.cc:654] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:51.059900 22564 tablet_bootstrap.cc:492] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f: Bootstrap starting.
I20260926 02:52:51.061609 22564 tablet_bootstrap.cc:654] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:51.063627 22563 tablet_bootstrap.cc:492] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376: No bootstrap required, opened a new log
I20260926 02:52:51.063788 22563 ts_tablet_manager.cc:1397] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260926 02:52:51.064247 22564 tablet_bootstrap.cc:492] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f: No bootstrap required, opened a new log
I20260926 02:52:51.064406 22564 ts_tablet_manager.cc:1397] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 02:52:51.064541 22563 raft_consensus.cc:348] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.064770 22563 raft_consensus.cc:374] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:51.064859 22563 raft_consensus.cc:729] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f265e515f6744cde815a3ca1b4932376, State: Initialized, Role: FOLLOWER
I20260926 02:52:51.065129 22563 consensus_queue.cc:260] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.065255 22564 raft_consensus.cc:348] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.065582 22564 raft_consensus.cc:374] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:51.065706 22564 raft_consensus.cc:729] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed52fbfc571b4439a2683dbbb31dcd9f, State: Initialized, Role: FOLLOWER
I20260926 02:52:51.066061 22563 ts_tablet_manager.cc:1428] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 02:52:51.066059 22564 consensus_queue.cc:260] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.066928 22564 ts_tablet_manager.cc:1428] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 02:52:51.091051 21945 catalog_manager.cc:4841] TS ed52fbfc571b4439a2683dbbb31dcd9f (127.19.185.193:36921): tablet 4710ad3a56cc4fc3a25f0e3ea42849d2 (table test-table [id=53a9037167564f9fb0ae5a145ab9bb9f]) successfully deleted
I20260926 02:52:51.091056 21948 catalog_manager.cc:4841] TS f265e515f6744cde815a3ca1b4932376 (127.19.185.194:37997): tablet 4710ad3a56cc4fc3a25f0e3ea42849d2 (table test-table [id=53a9037167564f9fb0ae5a145ab9bb9f]) successfully deleted
I20260926 02:52:51.103230 22375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 550ace0c7fea437b96d632a87000e5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:51.103264 22554 ts_tablet_manager.cc:1918] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:52:51.103463 22554 log.cc:1196] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/4710ad3a56cc4fc3a25f0e3ea42849d2
I20260926 02:52:51.103946 22554 ts_tablet_manager.cc:1939] T 4710ad3a56cc4fc3a25f0e3ea42849d2 P 7114a2baac384b11a4e89008cbc3e1eb: Deleting consensus metadata
I20260926 02:52:51.105261 21946 catalog_manager.cc:4841] TS 7114a2baac384b11a4e89008cbc3e1eb (127.19.185.195:40653): tablet 4710ad3a56cc4fc3a25f0e3ea42849d2 (table test-table [id=53a9037167564f9fb0ae5a145ab9bb9f]) successfully deleted
I20260926 02:52:51.108815 22568 tablet_bootstrap.cc:492] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb: Bootstrap starting.
I20260926 02:52:51.110167 22568 tablet_bootstrap.cc:654] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:51.112491 22568 tablet_bootstrap.cc:492] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb: No bootstrap required, opened a new log
I20260926 02:52:51.112663 22568 ts_tablet_manager.cc:1397] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:52:51.113511 22568 raft_consensus.cc:348] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.113737 22568 raft_consensus.cc:374] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:51.113838 22568 raft_consensus.cc:729] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7114a2baac384b11a4e89008cbc3e1eb, State: Initialized, Role: FOLLOWER
I20260926 02:52:51.114182 22568 consensus_queue.cc:260] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.114764 22568 ts_tablet_manager.cc:1428] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:52:51.214747 22533 raft_consensus.cc:482] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:52:51.214949 22533 raft_consensus.cc:504] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.215766 22533 leader_election.cc:288] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f265e515f6744cde815a3ca1b4932376 (127.19.185.194:37997), 7114a2baac384b11a4e89008cbc3e1eb (127.19.185.195:40653)
I20260926 02:52:51.216354 22395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550ace0c7fea437b96d632a87000e5cb" candidate_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7114a2baac384b11a4e89008cbc3e1eb" is_pre_election: true
I20260926 02:52:51.216364 22262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550ace0c7fea437b96d632a87000e5cb" candidate_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f265e515f6744cde815a3ca1b4932376" is_pre_election: true
I20260926 02:52:51.216614 22395 raft_consensus.cc:2454] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed52fbfc571b4439a2683dbbb31dcd9f in term 0.
I20260926 02:52:51.216627 22262 raft_consensus.cc:2454] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed52fbfc571b4439a2683dbbb31dcd9f in term 0.
I20260926 02:52:51.217128 22051 leader_election.cc:302] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7114a2baac384b11a4e89008cbc3e1eb, ed52fbfc571b4439a2683dbbb31dcd9f, f265e515f6744cde815a3ca1b4932376; no voters:
I20260926 02:52:51.217430 22533 raft_consensus.cc:2781] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:52:51.217533 22533 raft_consensus.cc:482] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:52:51.217592 22533 raft_consensus.cc:3037] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:51.304715 22533 raft_consensus.cc:504] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.305500 22533 leader_election.cc:288] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [CANDIDATE]: Term 1 election: Requested vote from peers f265e515f6744cde815a3ca1b4932376 (127.19.185.194:37997), 7114a2baac384b11a4e89008cbc3e1eb (127.19.185.195:40653)
I20260926 02:52:51.306141 22262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550ace0c7fea437b96d632a87000e5cb" candidate_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f265e515f6744cde815a3ca1b4932376"
I20260926 02:52:51.306191 22395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550ace0c7fea437b96d632a87000e5cb" candidate_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7114a2baac384b11a4e89008cbc3e1eb"
I20260926 02:52:51.306387 22262 raft_consensus.cc:3037] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:51.306407 22395 raft_consensus.cc:3037] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:51.308938 22395 raft_consensus.cc:2454] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed52fbfc571b4439a2683dbbb31dcd9f in term 1.
I20260926 02:52:51.309495 22049 leader_election.cc:302] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7114a2baac384b11a4e89008cbc3e1eb, ed52fbfc571b4439a2683dbbb31dcd9f; no voters:
I20260926 02:52:51.309849 22533 raft_consensus.cc:2781] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:51.310012 22533 raft_consensus.cc:686] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 LEADER]: Becoming Leader. State: Replica: ed52fbfc571b4439a2683dbbb31dcd9f, State: Running, Role: LEADER
I20260926 02:52:51.310285 22533 consensus_queue.cc:237] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.313344 21961 catalog_manager.cc:5495] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f reported cstate change: term changed from 0 to 1, leader changed from <none> to ed52fbfc571b4439a2683dbbb31dcd9f (127.19.185.193). New cstate: current_term: 1 leader_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } health_report { overall_health: UNKNOWN } } }
I20260926 02:52:51.365594 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:51.360262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.19.185.193:36921)
0926 02:52:51.360820 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:51.360825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:51.360844 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:52:51.360894 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:51.360897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:51.360955 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:51.361133 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:52:51.361593 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.362446 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.363892 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.364391 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.364491 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:51.364948 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:52:51.365045 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:52:51.365111 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:51.365297 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 02:52:51.365354 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":319,"threads_started":1}
I20260926 02:52:51.374481 22395 raft_consensus.cc:1261] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Refusing update from remote peer ed52fbfc571b4439a2683dbbb31dcd9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:51.374989 22533 consensus_queue.cc:1035] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:51.411580 22262 raft_consensus.cc:2454] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed52fbfc571b4439a2683dbbb31dcd9f in term 1.
I20260926 02:52:51.411950 22261 raft_consensus.cc:1261] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Refusing update from remote peer ed52fbfc571b4439a2683dbbb31dcd9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:51.413193 22538 consensus_queue.cc:1035] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:51.420710 21960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:52:51.420980 21960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:52:51.423813 21960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76: Sending DeleteTablet for 3 replicas of tablet 550ace0c7fea437b96d632a87000e5cb
I20260926 02:52:51.424567 22237 tablet_service.cc:1514] Processing DeleteTablet for tablet 550ace0c7fea437b96d632a87000e5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:51 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:34502
I20260926 02:52:51.424573 22097 tablet_service.cc:1514] Processing DeleteTablet for tablet 550ace0c7fea437b96d632a87000e5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:51 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:41616
I20260926 02:52:51.424853 22553 tablet_replica.cc:330] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376: stopping tablet replica
I20260926 02:52:51.424800 22375 tablet_service.cc:1514] Processing DeleteTablet for tablet 550ace0c7fea437b96d632a87000e5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:51 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:55256
I20260926 02:52:51.424925 22552 tablet_replica.cc:330] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f: stopping tablet replica
I20260926 02:52:51.425076 22553 raft_consensus.cc:2229] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:52:51.425084 22554 tablet_replica.cc:330] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb: stopping tablet replica
I20260926 02:52:51.425117 22552 raft_consensus.cc:2229] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:52:51.425289 22554 raft_consensus.cc:2229] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:52:51.425324 22553 raft_consensus.cc:2258] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:51.425500 22552 raft_consensus.cc:2258] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:51.425545 22554 raft_consensus.cc:2258] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:51.426515 22553 ts_tablet_manager.cc:1905] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:51.426548 22552 ts_tablet_manager.cc:1905] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:51.426594 21960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 02:52:51.426884 22554 ts_tablet_manager.cc:1905] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 02:52:51.427241 21960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:52:51.434576 22097 tablet_service.cc:1467] Processing CreateTablet for tablet 73330d8d15bc4b7e99d7f2e8a0a7fdd7 (DEFAULT_TABLE table=test-table [id=29e2d6d1212e4c67b0a7da89af7133cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:51.434716 22237 tablet_service.cc:1467] Processing CreateTablet for tablet 73330d8d15bc4b7e99d7f2e8a0a7fdd7 (DEFAULT_TABLE table=test-table [id=29e2d6d1212e4c67b0a7da89af7133cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:51.435019 22375 tablet_service.cc:1467] Processing CreateTablet for tablet 73330d8d15bc4b7e99d7f2e8a0a7fdd7 (DEFAULT_TABLE table=test-table [id=29e2d6d1212e4c67b0a7da89af7133cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:52:51.510673 22237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73330d8d15bc4b7e99d7f2e8a0a7fdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:51.511404 22097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73330d8d15bc4b7e99d7f2e8a0a7fdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:51.513794 22375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73330d8d15bc4b7e99d7f2e8a0a7fdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:51.514380 22553 ts_tablet_manager.cc:1918] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:52:51.514528 22553 log.cc:1196] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/550ace0c7fea437b96d632a87000e5cb
I20260926 02:52:51.514907 22553 ts_tablet_manager.cc:1939] T 550ace0c7fea437b96d632a87000e5cb P f265e515f6744cde815a3ca1b4932376: Deleting consensus metadata
I20260926 02:52:51.518803 22563 tablet_bootstrap.cc:492] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376: Bootstrap starting.
I20260926 02:52:51.520781 22563 tablet_bootstrap.cc:654] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:51.520789 22568 tablet_bootstrap.cc:492] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb: Bootstrap starting.
I20260926 02:52:51.522377 22568 tablet_bootstrap.cc:654] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:51.522919 22564 tablet_bootstrap.cc:492] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f: Bootstrap starting.
I20260926 02:52:51.524771 22564 tablet_bootstrap.cc:654] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:51.524969 22568 tablet_bootstrap.cc:492] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb: No bootstrap required, opened a new log
I20260926 02:52:51.524995 22563 tablet_bootstrap.cc:492] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376: No bootstrap required, opened a new log
I20260926 02:52:51.525149 22568 ts_tablet_manager.cc:1397] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260926 02:52:51.525146 22563 ts_tablet_manager.cc:1397] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260926 02:52:51.525744 22568 raft_consensus.cc:348] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.525997 22568 raft_consensus.cc:374] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:51.526096 22568 raft_consensus.cc:729] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7114a2baac384b11a4e89008cbc3e1eb, State: Initialized, Role: FOLLOWER
I20260926 02:52:51.526013 22563 raft_consensus.cc:348] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.526317 22563 raft_consensus.cc:374] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:51.526434 22563 raft_consensus.cc:729] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f265e515f6744cde815a3ca1b4932376, State: Initialized, Role: FOLLOWER
I20260926 02:52:51.526353 22568 consensus_queue.cc:260] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.526706 22563 consensus_queue.cc:260] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.526944 22568 ts_tablet_manager.cc:1428] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 02:52:51.527173 22563 ts_tablet_manager.cc:1428] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 02:52:51.528064 22564 tablet_bootstrap.cc:492] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f: No bootstrap required, opened a new log
I20260926 02:52:51.528240 22564 ts_tablet_manager.cc:1397] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 02:52:51.528865 22564 raft_consensus.cc:348] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.529085 22564 raft_consensus.cc:374] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:51.529183 22564 raft_consensus.cc:729] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed52fbfc571b4439a2683dbbb31dcd9f, State: Initialized, Role: FOLLOWER
I20260926 02:52:51.529421 22564 consensus_queue.cc:260] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.530009 22564 ts_tablet_manager.cc:1428] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 02:52:51.557324 21948 catalog_manager.cc:4841] TS f265e515f6744cde815a3ca1b4932376 (127.19.185.194:37997): tablet 550ace0c7fea437b96d632a87000e5cb (table test-table [id=66883228173f49d48a216dff74cb1499]) successfully deleted
I20260926 02:52:51.604000 22567 raft_consensus.cc:482] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:52:51.604225 22567 raft_consensus.cc:504] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.605966 22567 leader_election.cc:288] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed52fbfc571b4439a2683dbbb31dcd9f (127.19.185.193:36921), 7114a2baac384b11a4e89008cbc3e1eb (127.19.185.195:40653)
I20260926 02:52:51.614621 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:51.606422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:33801 (local address 127.19.185.193:36921)
0926 02:52:51.606649 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:51.606655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:51.607356 (+ 701us) server_negotiation.cc:407] Connection header received
0926 02:52:51.607692 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:51.607701 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:51.607798 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:51.607887 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:51.608420 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.609464 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.612185 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.613424 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.613590 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:51.613706 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:51.614229 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 02:52:51.614361 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:52:51.614717 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:51.606498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:35287 (local address 127.19.185.195:40653)
0926 02:52:51.608065 (+ 1567us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:51.608070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:51.608089 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:52:51.608149 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:51.608153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:51.608211 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:51.608331 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:52:51.608876 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.609957 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.612400 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.613590 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.613737 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:51.613804 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:51.614229 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 02:52:51.614413 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1429,"thread_start_us":1275,"threads_started":1}
I20260926 02:52:51.614894 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:51.606309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:40653 (local address 127.19.185.194:35287)
0926 02:52:51.607775 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:51.607799 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:51.607925 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:51.608360 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:51.608366 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:51.608392 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:51.608720 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:51.608727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.610085 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.610092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:51.612263 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:51.612273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.613822 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.613827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:51.613997 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:51.614083 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 02:52:51.614180 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 02:52:51.614366 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1281,"mutex_wait_us":118,"thread_start_us":990,"threads_started":1}
I20260926 02:52:51.615453 22579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:51.606143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:36921 (local address 127.19.185.194:33801)
0926 02:52:51.607206 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:51.607235 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:51.607526 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:51.607943 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:51.607949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:51.607976 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:52:51.608248 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:51.608256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.609607 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.609638 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:51.612023 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:51.612038 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.613728 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.613738 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:51.613929 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:51.614082 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 02:52:51.614180 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 02:52:51.614366 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":212,"thread_start_us":701,"threads_started":1}
I20260926 02:52:51.615667 22395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73330d8d15bc4b7e99d7f2e8a0a7fdd7" candidate_uuid: "f265e515f6744cde815a3ca1b4932376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7114a2baac384b11a4e89008cbc3e1eb" is_pre_election: true
I20260926 02:52:51.615872 22395 raft_consensus.cc:2454] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f265e515f6744cde815a3ca1b4932376 in term 0.
I20260926 02:52:51.616089 22123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73330d8d15bc4b7e99d7f2e8a0a7fdd7" candidate_uuid: "f265e515f6744cde815a3ca1b4932376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" is_pre_election: true
I20260926 02:52:51.616364 22123 raft_consensus.cc:2454] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f265e515f6744cde815a3ca1b4932376 in term 0.
I20260926 02:52:51.616384 22189 leader_election.cc:302] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7114a2baac384b11a4e89008cbc3e1eb, f265e515f6744cde815a3ca1b4932376; no voters:
I20260926 02:52:51.616597 22554 ts_tablet_manager.cc:1918] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:52:51.616755 22554 log.cc:1196] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/550ace0c7fea437b96d632a87000e5cb
I20260926 02:52:51.616837 22567 raft_consensus.cc:2781] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:52:51.616958 22567 raft_consensus.cc:482] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:52:51.617056 22567 raft_consensus.cc:3037] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:51.617177 22554 ts_tablet_manager.cc:1939] T 550ace0c7fea437b96d632a87000e5cb P 7114a2baac384b11a4e89008cbc3e1eb: Deleting consensus metadata
I20260926 02:52:51.618145 21946 catalog_manager.cc:4841] TS 7114a2baac384b11a4e89008cbc3e1eb (127.19.185.195:40653): tablet 550ace0c7fea437b96d632a87000e5cb (table test-table [id=66883228173f49d48a216dff74cb1499]) successfully deleted
I20260926 02:52:51.619606 22552 ts_tablet_manager.cc:1918] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:52:51.619794 22552 log.cc:1196] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/550ace0c7fea437b96d632a87000e5cb
I20260926 02:52:51.620167 22552 ts_tablet_manager.cc:1939] T 550ace0c7fea437b96d632a87000e5cb P ed52fbfc571b4439a2683dbbb31dcd9f: Deleting consensus metadata
I20260926 02:52:51.620510 22567 raft_consensus.cc:504] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.621289 22567 leader_election.cc:288] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [CANDIDATE]: Term 1 election: Requested vote from peers ed52fbfc571b4439a2683dbbb31dcd9f (127.19.185.193:36921), 7114a2baac384b11a4e89008cbc3e1eb (127.19.185.195:40653)
I20260926 02:52:51.621714 22123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73330d8d15bc4b7e99d7f2e8a0a7fdd7" candidate_uuid: "f265e515f6744cde815a3ca1b4932376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f"
I20260926 02:52:51.621714 22395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73330d8d15bc4b7e99d7f2e8a0a7fdd7" candidate_uuid: "f265e515f6744cde815a3ca1b4932376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7114a2baac384b11a4e89008cbc3e1eb"
I20260926 02:52:51.621920 22395 raft_consensus.cc:3037] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:51.621920 22123 raft_consensus.cc:3037] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:51.674799 21945 catalog_manager.cc:4841] TS ed52fbfc571b4439a2683dbbb31dcd9f (127.19.185.193:36921): tablet 550ace0c7fea437b96d632a87000e5cb (table test-table [id=66883228173f49d48a216dff74cb1499]) successfully deleted
I20260926 02:52:51.715216 22395 raft_consensus.cc:2454] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f265e515f6744cde815a3ca1b4932376 in term 1.
I20260926 02:52:51.715211 22123 raft_consensus.cc:2454] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f265e515f6744cde815a3ca1b4932376 in term 1.
I20260926 02:52:51.715931 22188 leader_election.cc:302] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7114a2baac384b11a4e89008cbc3e1eb, ed52fbfc571b4439a2683dbbb31dcd9f, f265e515f6744cde815a3ca1b4932376; no voters:
I20260926 02:52:51.716348 22567 raft_consensus.cc:2781] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:51.716897 22567 raft_consensus.cc:686] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [term 1 LEADER]: Becoming Leader. State: Replica: f265e515f6744cde815a3ca1b4932376, State: Running, Role: LEADER
I20260926 02:52:51.717372 22567 consensus_queue.cc:237] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } }
I20260926 02:52:51.721340 21960 catalog_manager.cc:5495] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 reported cstate change: term changed from 0 to 1, leader changed from <none> to f265e515f6744cde815a3ca1b4932376 (127.19.185.194). New cstate: current_term: 1 leader_uuid: "f265e515f6744cde815a3ca1b4932376" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f265e515f6744cde815a3ca1b4932376" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 37997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 } health_report { overall_health: UNKNOWN } } }
I20260926 02:52:51.751546 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:51.745642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.19.185.194:37997)
0926 02:52:51.746145 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:51.746150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:51.746169 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:52:51.746290 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:51.746293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:51.746365 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:51.746506 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:52:51.746837 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.747740 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.749091 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.749525 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.749626 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:51.750023 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:52:51.750769 (+ 746us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:52:51.750937 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:51.751015 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 02:52:51.751075 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":273,"threads_started":1}
I20260926 02:52:51.768088 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:51.762055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:43107 (local address 127.19.185.193:36921)
0926 02:52:51.762297 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:51.762305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:51.762908 (+ 603us) server_negotiation.cc:407] Connection header received
0926 02:52:51.763146 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:51.763150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:51.763210 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:51.763321 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:52:51.763948 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.764724 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.766263 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.766675 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.766852 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:51.767341 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:52:51.767456 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:52:51.767567 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:51.767673 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 02:52:51.767758 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:52:51.768224 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:51.762041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:36921 (local address 127.19.185.194:43107)
0926 02:52:51.762808 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:51.762830 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:51.762982 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:51.763342 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:51.763347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:51.763359 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:52:51.763797 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:51.763807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.764885 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.764894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:51.766123 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:51.766135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.766934 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.766941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:51.767064 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:51.767117 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:52:51.767556 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:52:51.767566 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:52:51.767655 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 02:52:51.767784 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":348,"threads_started":1}
I20260926 02:52:51.776711 22395 raft_consensus.cc:1261] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Refusing update from remote peer f265e515f6744cde815a3ca1b4932376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:51.776924 22123 raft_consensus.cc:1261] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Refusing update from remote peer f265e515f6744cde815a3ca1b4932376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:52:51.777516 22582 consensus_queue.cc:1035] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7114a2baac384b11a4e89008cbc3e1eb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:51.777881 22567 consensus_queue.cc:1035] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed52fbfc571b4439a2683dbbb31dcd9f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 36921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:52:51.806301 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:51.800243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:35007 (local address 127.19.185.194:37997)
0926 02:52:51.800441 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:51.800449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:51.801090 (+ 641us) server_negotiation.cc:407] Connection header received
0926 02:52:51.801259 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:51.801264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:51.801377 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:51.801475 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:52:51.802045 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.802754 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.804301 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.804936 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.805058 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:51.805594 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:52:51.805712 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:52:51.805792 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:51.805989 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 02:52:51.806049 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:52:51.806470 22593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:51.800139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:37997 (local address 127.19.185.193:35007)
0926 02:52:51.800966 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:51.800986 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:51.801141 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:51.801531 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:51.801538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:51.801554 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:52:51.801883 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:51.801892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.802889 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.802897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:51.804182 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:51.804199 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:51.805211 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:51.805219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:51.805390 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:51.805428 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:52:51.805868 (+ 440us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:52:51.805888 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 02:52:51.805965 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 02:52:51.806072 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":486,"threads_started":1}
W20260926 02:52:51.807737 22535 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 02:52:53.410799 21960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:52:53.411012 21960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:52:53.415095 21960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56272cc72b814f7b908d32e53faa2d76: Sending DeleteTablet for 3 replicas of tablet 73330d8d15bc4b7e99d7f2e8a0a7fdd7
I20260926 02:52:53.416030 22237 tablet_service.cc:1514] Processing DeleteTablet for tablet 73330d8d15bc4b7e99d7f2e8a0a7fdd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:53 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:34502
I20260926 02:52:53.416149 22097 tablet_service.cc:1514] Processing DeleteTablet for tablet 73330d8d15bc4b7e99d7f2e8a0a7fdd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:53 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:41616
I20260926 02:52:53.416741 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 22032
I20260926 02:52:53.417506 22375 tablet_service.cc:1514] Processing DeleteTablet for tablet 73330d8d15bc4b7e99d7f2e8a0a7fdd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:52:53 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:55256
I20260926 02:52:53.418270 22617 tablet_replica.cc:330] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376: stopping tablet replica
I20260926 02:52:53.418406 22617 raft_consensus.cc:2229] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:52:53.418901 22617 raft_consensus.cc:2258] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:53.420243 22617 ts_tablet_manager.cc:1905] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P f265e515f6744cde815a3ca1b4932376: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:53.426092 22619 tablet_replica.cc:330] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb: stopping tablet replica
I20260926 02:52:53.426338 22619 raft_consensus.cc:2229] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:52:53.426615 22619 raft_consensus.cc:2258] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:53.427227 22618 tablet_replica.cc:330] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f: stopping tablet replica
I20260926 02:52:53.427471 22618 raft_consensus.cc:2229] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:52:53.427743 22618 raft_consensus.cc:2258] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:52:53.428112 22619 ts_tablet_manager.cc:1905] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:52:53.428699 22618 ts_tablet_manager.cc:1905] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P ed52fbfc571b4439a2683dbbb31dcd9f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 02:52:53 dist-test-slave-ktng krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391170, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:52:53.440771 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:53.418833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43144 (local address 127.19.185.193:36921)
0926 02:52:53.425227 (+ 6394us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:53.425231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:53.425251 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:52:53.425299 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:53.425302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:53.425367 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:53.425501 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:53.427238 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:53.428456 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:53.430294 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:53.430820 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:53.430923 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:53.435024 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:53.435064 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:53.435090 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:53.435131 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:53.437955 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:53.438571 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:53.438588 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:53.438590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:53.438732 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:53.439157 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:53.439163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:53.439168 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:53.439578 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:53.439884 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:53.440206 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 02:52:53.440436 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6262,"thread_start_us":6131,"threads_started":1}
W20260926 02:52:53.441402 22163 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43144
W20260926 02:52:53.442062 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 22032: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:52:53.442308 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 22032
I20260926 02:52:53.504236 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 22172
W20260926 02:52:53.504166 21945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.185.193:36921 (error 108)
I20260926 02:52:53.507678 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:53.505881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:36921 (local address 127.0.0.1:43146)
0926 02:52:53.506982 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:53.507334 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":799,"threads_started":1}
W20260926 02:52:53.507956 21945 catalog_manager.cc:4584] TS ed52fbfc571b4439a2683dbbb31dcd9f (127.19.185.193:36921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73330d8d15bc4b7e99d7f2e8a0a7fdd7: Network error: Client connection negotiation failed: client connection to 127.19.185.193:36921: connect: Connection refused (error 111)
Sep 26 02:52:53 dist-test-slave-ktng krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391170, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:52:53.522811 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:53.505379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.19.185.194:37997)
0926 02:52:53.506087 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:53.506091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:53.506107 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:52:53.506171 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:53.506174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:53.506243 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:53.506369 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:53.508195 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:53.509350 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:53.510647 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:53.511316 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:53.511436 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:53.516508 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:53.516523 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:53.516539 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:53.516577 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:53.520245 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:53.520909 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:53.520916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:53.520918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:53.521050 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:53.521392 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:53.521395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:53.521397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:53.521737 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:53.521989 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:53.522305 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 02:52:53.522471 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":485,"threads_started":1}
W20260926 02:52:53.523614 22302 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35484
W20260926 02:52:53.524220 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 22172: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:52:53.524356 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 22172
I20260926 02:52:53.561754 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:53.561245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:36921 (local address 127.0.0.1:43148)
0926 02:52:53.561481 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:53.561599 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 02:52:53.562865 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:53.562491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:36921 (local address 127.0.0.1:43152)
0926 02:52:53.562645 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:53.562734 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 02:52:53.578096 22619 ts_tablet_manager.cc:1918] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:52:53.578398 22619 log.cc:1196] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/73330d8d15bc4b7e99d7f2e8a0a7fdd7
I20260926 02:52:53.578876 22619 ts_tablet_manager.cc:1939] T 73330d8d15bc4b7e99d7f2e8a0a7fdd7 P 7114a2baac384b11a4e89008cbc3e1eb: Deleting consensus metadata
I20260926 02:52:53.583953 21946 catalog_manager.cc:4841] TS 7114a2baac384b11a4e89008cbc3e1eb (127.19.185.195:40653): tablet 73330d8d15bc4b7e99d7f2e8a0a7fdd7 (table test-table [id=29e2d6d1212e4c67b0a7da89af7133cf]) successfully deleted
I20260926 02:52:53.600873 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 22311
I20260926 02:52:53.601480 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:53.600983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:37997 (local address 127.0.0.1:35492)
0926 02:52:53.601185 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:53.601283 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.194:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
Sep 26 02:52:53 dist-test-slave-ktng krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391170, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:52:53.616835 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:53.601686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.19.185.195:40653)
0926 02:52:53.602183 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:53.602188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:53.602204 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:52:53.602257 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:53.602261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:53.602317 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:53.602448 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:53.603890 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:53.604599 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:53.605900 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:53.606356 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:53.606459 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:53.610933 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:53.610949 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:53.610966 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:53.611007 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:53.614055 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:53.614702 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:53.614709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:53.614712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:53.614833 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:53.615168 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:53.615172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:53.615175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:53.615659 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:53.615874 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:53.616238 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 02:52:53.616441 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":262,"threads_started":1}
W20260926 02:52:53.617587 22441 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43060
W20260926 02:52:53.618510 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 22311: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:52:53.618700 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 22311
I20260926 02:52:53.623548 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:53.622907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:36921 (local address 127.0.0.1:43162)
0926 02:52:53.623222 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:53.623341 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260926 02:52:53.624537 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:53.624208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:36921 (local address 127.0.0.1:43166)
0926 02:52:53.624358 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:53.624429 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 02:52:53.633973 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:53.633420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:37997 (local address 127.0.0.1:35504)
0926 02:52:53.633698 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:53.633831 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.194:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260926 02:52:53.635135 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:53.634742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:37997 (local address 127.0.0.1:35514)
0926 02:52:53.634943 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:53.635025 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.194:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 02:52:53.643679 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 21929
I20260926 02:52:53.644802 22001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:52:53.760917 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:53.759913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:37997 (local address 127.0.0.1:35528)
0926 02:52:53.760346 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:53.760500 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.194:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266}
I20260926 02:52:53.760921 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 21929
2026-09-26T02:52:53Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9012 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:52:54 dist-test-slave-ktng krb5kdc[22630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:52:54 dist-test-slave-ktng krb5kdc[22630](info): set up 2 sockets
Sep 26 02:52:54 dist-test-slave-ktng krb5kdc[22630](info): commencing operation
krb5kdc: starting...
W20260926 02:52:55.136595 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.330s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:52:55 dist-test-slave-ktng krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:52:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:52:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:52:55.340201 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:34979
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:34979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:55.676455 22643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:55.676784 22643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:55.676949 22643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:55.684230 22643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:52:55.684374 22643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:55.684479 22643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:55.684583 22643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:52:55.684667 22643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:52:55.694440 22643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:34979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:34979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:55.695900 22643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:55.697365 22643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:55.709697 22648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:55.709725 22649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:55.709748 22651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:55.711162 22643 server_base.cc:1034] running on GCE node
I20260926 02:52:55.712003 22643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:55.713487 22643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:55.714725 22643 hybrid_clock.cc:648] HybridClock initialized: now 1790391175714673 us; error 44 us; skew 500 ppm
I20260926 02:52:55.715068 22643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:55 dist-test-slave-ktng krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391175, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:55.718322 22643 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:52:55.719974 22643 webserver.cc:460] Webserver started at http://127.19.185.254:38751/ using document root <none> and password file <none>
I20260926 02:52:55.720669 22643 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:55.720808 22643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:55.721118 22643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:55.724200 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "4333875421be42f18cbd96284d76d8ae"
format_stamp: "Formatted at 2026-09-26 02:52:55 on dist-test-slave-ktng"
server_key: "40bba196b5e1d223b793aee8f92160a1"
server_key_iv: "aa9246f3e0c1039fbfca191628a83d95"
server_key_version: "encryptionkey@0"
I20260926 02:52:55.724938 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "4333875421be42f18cbd96284d76d8ae"
format_stamp: "Formatted at 2026-09-26 02:52:55 on dist-test-slave-ktng"
server_key: "40bba196b5e1d223b793aee8f92160a1"
server_key_iv: "aa9246f3e0c1039fbfca191628a83d95"
server_key_version: "encryptionkey@0"
I20260926 02:52:55.730378 22643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:52:55.734376 22658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:55.735867 22643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:52:55.736045 22643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "4333875421be42f18cbd96284d76d8ae"
format_stamp: "Formatted at 2026-09-26 02:52:55 on dist-test-slave-ktng"
server_key: "40bba196b5e1d223b793aee8f92160a1"
server_key_iv: "aa9246f3e0c1039fbfca191628a83d95"
server_key_version: "encryptionkey@0"
I20260926 02:52:55.736254 22643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:55.749111 22643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:55.753443 22643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:55.753739 22643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:55.772673 22643 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:34979
I20260926 02:52:55.772696 22709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:34979 every 8 connection(s)
I20260926 02:52:55.774123 22643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:52:55.778209 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 22643
I20260926 02:52:55.778307 22710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:55.778458 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:52:55.779241 20199 external_mini_cluster.cc:1428] Setting key 6a918bbc9fcbf8099db984c2d30b4a8b
I20260926 02:52:55.792645 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae: Bootstrap starting.
Sep 26 02:52:55 dist-test-slave-ktng krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:55.796491 22710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:55.797976 22710 log.cc:824] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae: Log is configured to *not* fsync() on all Append() calls
I20260926 02:52:55.801353 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae: No bootstrap required, opened a new log
I20260926 02:52:55.804013 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:55.781406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.19.185.254:34979)
0926 02:52:55.782239 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:55.782251 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:55.782293 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:52:55.784506 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:55.784532 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:55.784924 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:55.785262 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:55.787012 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:55.788372 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:55.790196 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:55.790900 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:55.791045 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:55.795679 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:55.795689 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:55.795712 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:55.795769 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:55.799541 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:55.800205 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:55.800213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:55.800222 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:55.800420 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:55.800766 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:55.800770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:55.800773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:55.801458 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:55.801752 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:55.802107 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 02:52:55.802520 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":421,"threads_started":1}
I20260926 02:52:55.810158 22710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333875421be42f18cbd96284d76d8ae" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34979 } }
I20260926 02:52:55.810484 22710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:55.810557 22710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4333875421be42f18cbd96284d76d8ae, State: Initialized, Role: FOLLOWER
I20260926 02:52:55.811108 22710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333875421be42f18cbd96284d76d8ae" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34979 } }
I20260926 02:52:55.811359 22710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:52:55.811477 22710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:52:55.811600 22710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:55.813896 22710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333875421be42f18cbd96284d76d8ae" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34979 } }
I20260926 02:52:55.814327 22710 leader_election.cc:302] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4333875421be42f18cbd96284d76d8ae; no voters:
I20260926 02:52:55.814883 22710 leader_election.cc:288] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:52:55.815027 22715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:55.815779 22715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [term 1 LEADER]: Becoming Leader. State: Replica: 4333875421be42f18cbd96284d76d8ae, State: Running, Role: LEADER
I20260926 02:52:55.816288 22715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333875421be42f18cbd96284d76d8ae" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34979 } }
I20260926 02:52:55.817122 22710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:52:55.819151 22716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4333875421be42f18cbd96284d76d8ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333875421be42f18cbd96284d76d8ae" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34979 } } }
I20260926 02:52:55.819489 22716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [sys.catalog]: This master's current role is: LEADER
I20260926 02:52:55.819888 22717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4333875421be42f18cbd96284d76d8ae. Latest consensus state: current_term: 1 leader_uuid: "4333875421be42f18cbd96284d76d8ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333875421be42f18cbd96284d76d8ae" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34979 } } }
I20260926 02:52:55.820199 22717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae [sys.catalog]: This master's current role is: LEADER
I20260926 02:52:55.821823 22722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:52:55.827896 22722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:52:55.836725 22722 catalog_manager.cc:1347] Generated new cluster ID: 0a1a413500134a54a713b5c40e4ad9e7
I20260926 02:52:55.836844 22722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:52:55.864120 22722 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:52:55.865583 22722 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:52:55.881191 22722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4333875421be42f18cbd96284d76d8ae: Generated new TSK 0
I20260926 02:52:55.882138 22722 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:52:55.932701 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34979
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:56.265492 22736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:56.265818 22736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:56.265995 22736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:56.273356 22736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:56.273505 22736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:56.273679 22736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:52:56.283646 22736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:56.284924 22736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:56.286269 22736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:56.299679 22741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:56.299669 22742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:56.300062 22744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:56.300499 22736 server_base.cc:1034] running on GCE node
I20260926 02:52:56.301054 22736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:56.301950 22736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:56.303194 22736 hybrid_clock.cc:648] HybridClock initialized: now 1790391176303127 us; error 55 us; skew 500 ppm
I20260926 02:52:56.303548 22736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:56 dist-test-slave-ktng krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391176, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:56.312327 22736 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:52:56.313791 22736 webserver.cc:460] Webserver started at http://127.19.185.193:34993/ using document root <none> and password file <none>
I20260926 02:52:56.314497 22736 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:56.314637 22736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:56.314949 22736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:56.318006 22736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "0a8ba99129a44f10bf347dbef39fcd5f"
format_stamp: "Formatted at 2026-09-26 02:52:56 on dist-test-slave-ktng"
server_key: "0625ac28cbb280e124856c7e7ff0885b"
server_key_iv: "c7ae9d0e46375e01d0ebab1e51e79f98"
server_key_version: "encryptionkey@0"
I20260926 02:52:56.318840 22736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "0a8ba99129a44f10bf347dbef39fcd5f"
format_stamp: "Formatted at 2026-09-26 02:52:56 on dist-test-slave-ktng"
server_key: "0625ac28cbb280e124856c7e7ff0885b"
server_key_iv: "c7ae9d0e46375e01d0ebab1e51e79f98"
server_key_version: "encryptionkey@0"
I20260926 02:52:56.324182 22736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:52:56.327730 22751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:56.329201 22736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:52:56.329367 22736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "0a8ba99129a44f10bf347dbef39fcd5f"
format_stamp: "Formatted at 2026-09-26 02:52:56 on dist-test-slave-ktng"
server_key: "0625ac28cbb280e124856c7e7ff0885b"
server_key_iv: "c7ae9d0e46375e01d0ebab1e51e79f98"
server_key_version: "encryptionkey@0"
I20260926 02:52:56.329567 22736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:56.342356 22736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:56.346694 22736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:56.346979 22736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:56.348001 22736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:52:56.349359 22736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:56.349463 22736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:56.349603 22736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:56.349674 22736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:56.382768 22736 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:46579
I20260926 02:52:56.382813 22863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:46579 every 8 connection(s)
I20260926 02:52:56.384181 22736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:52:56.394001 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 22736
I20260926 02:52:56.394255 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:52:56.394917 20199 external_mini_cluster.cc:1428] Setting key 2c0f8602e198aacb0eaf465455daa271
Sep 26 02:52:56 dist-test-slave-ktng krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391176, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:56.406163 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:56.386828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:40703 (local address 127.19.185.254:34979)
0926 02:52:56.387257 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:56.387262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:56.388623 (+ 1361us) server_negotiation.cc:407] Connection header received
0926 02:52:56.390460 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:56.390466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:56.390555 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:56.390672 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:56.392219 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:56.392972 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:56.394673 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:56.395134 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:56.395279 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:56.399277 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:56.399287 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:56.399315 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:56.399363 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:56.402761 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:56.403786 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:56.403790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:56.403792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:56.403903 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:56.404380 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:56.404383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:56.404385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:56.404640 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:56.404746 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:56.405709 (+ 963us) server_negotiation.cc:299] Negotiation successful
0926 02:52:56.405874 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":235,"threads_started":1}
I20260926 02:52:56.407881 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:56.387602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:34979 (local address 127.19.185.193:40703)
0926 02:52:56.388346 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:56.388394 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:56.390192 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:56.390898 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:56.390906 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:56.391509 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:56.392057 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:56.392074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:56.393124 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:56.393136 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:56.394533 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:56.394546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:56.395393 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:56.395402 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:56.395651 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:56.396369 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:56.396395 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:56.399030 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:56.402911 (+ 3881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:56.402924 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:56.402943 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:56.403654 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:56.404015 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:56.404020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:56.404023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:56.404267 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:56.404761 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:56.404773 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:56.405220 (+ 447us) client_negotiation.cc:769] Sending connection context
0926 02:52:56.405631 (+ 411us) client_negotiation.cc:240] Negotiation successful
0926 02:52:56.406256 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":314,"threads_started":1}
I20260926 02:52:56.409471 22864 heartbeater.cc:347] Connected to a master server at 127.19.185.254:34979
I20260926 02:52:56.409789 22864 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:56.410498 22864 heartbeater.cc:511] Master 127.19.185.254:34979 requested a full tablet report, sending...
I20260926 02:52:56.412720 22675 ts_manager.cc:194] Registered new tserver with Master: 0a8ba99129a44f10bf347dbef39fcd5f (127.19.185.193:46579)
I20260926 02:52:56.414165 22675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:40703
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:52:56.440974 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34979
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:56.779692 22870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:56.780041 22870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:56.780205 22870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:56.787621 22870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:56.787773 22870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:56.787981 22870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:52:56.798110 22870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:56.799628 22870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:56.801017 22870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:56.814095 22875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:56.814064 22878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:56.814141 22876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:56.814845 22870 server_base.cc:1034] running on GCE node
I20260926 02:52:56.815541 22870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:56.816461 22870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:56.817700 22870 hybrid_clock.cc:648] HybridClock initialized: now 1790391176817654 us; error 57 us; skew 500 ppm
I20260926 02:52:56.818076 22870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:56 dist-test-slave-ktng krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391176, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:56.821357 22870 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:52:56.823567 22870 webserver.cc:460] Webserver started at http://127.19.185.194:41875/ using document root <none> and password file <none>
I20260926 02:52:56.824260 22870 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:56.824393 22870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:56.824710 22870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:56.827610 22870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "739d34d289f54e52aebca0738d3106a9"
format_stamp: "Formatted at 2026-09-26 02:52:56 on dist-test-slave-ktng"
server_key: "597ff5c2c30432a15c4d2f9fc055a3d6"
server_key_iv: "691286705c917866f17503d53a04d331"
server_key_version: "encryptionkey@0"
I20260926 02:52:56.828373 22870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "739d34d289f54e52aebca0738d3106a9"
format_stamp: "Formatted at 2026-09-26 02:52:56 on dist-test-slave-ktng"
server_key: "597ff5c2c30432a15c4d2f9fc055a3d6"
server_key_iv: "691286705c917866f17503d53a04d331"
server_key_version: "encryptionkey@0"
I20260926 02:52:56.833582 22870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:52:56.837134 22885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:56.838685 22870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:52:56.838886 22870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "739d34d289f54e52aebca0738d3106a9"
format_stamp: "Formatted at 2026-09-26 02:52:56 on dist-test-slave-ktng"
server_key: "597ff5c2c30432a15c4d2f9fc055a3d6"
server_key_iv: "691286705c917866f17503d53a04d331"
server_key_version: "encryptionkey@0"
I20260926 02:52:56.839102 22870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:56.859560 22870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:56.864122 22870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:56.864499 22870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:56.865644 22870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:52:56.867125 22870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:56.867199 22870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:56.867285 22870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:56.867347 22870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 02:52:56.900866 22870 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:38085
I20260926 02:52:56.900951 22997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:38085 every 8 connection(s)
I20260926 02:52:56.902192 22870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:52:56.909906 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 22870
I20260926 02:52:56.910097 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:52:56.910631 20199 external_mini_cluster.cc:1428] Setting key 7355dfe8e92e188b766705b5ea7f89fc
Sep 26 02:52:56 dist-test-slave-ktng krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391176, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:56.925532 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:56.904754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:54859 (local address 127.19.185.254:34979)
0926 02:52:56.904975 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:56.904981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:56.906459 (+ 1478us) server_negotiation.cc:407] Connection header received
0926 02:52:56.908398 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:56.908403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:56.908491 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:56.908582 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:56.910530 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:56.911456 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:56.913415 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:56.913844 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:56.913946 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:56.918311 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:56.918319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:56.918326 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:56.918372 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:56.921978 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:56.922759 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:56.922768 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:56.922774 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:56.922953 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:56.923486 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:56.923494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:56.923501 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:56.923902 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:56.924048 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:56.924799 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 02:52:56.925077 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:52:56.926261 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:56.905288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:34979 (local address 127.19.185.194:54859)
0926 02:52:56.906253 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:56.906307 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:56.908125 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:56.908858 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:56.908875 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:56.909445 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:56.910323 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:56.910349 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:56.911603 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:56.911610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:56.913248 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:56.913266 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:56.914096 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:56.914106 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:56.914471 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:56.915185 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:56.915225 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:56.918068 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:56.922148 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:56.922164 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:56.922179 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:56.922649 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:56.923073 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:56.923077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:56.923079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:56.923376 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:56.924076 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:56.924082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:56.924387 (+ 305us) client_negotiation.cc:769] Sending connection context
0926 02:52:56.924626 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 02:52:56.925017 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":327,"threads_started":1}
I20260926 02:52:56.927901 22998 heartbeater.cc:347] Connected to a master server at 127.19.185.254:34979
I20260926 02:52:56.928246 22998 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:56.929147 22998 heartbeater.cc:511] Master 127.19.185.254:34979 requested a full tablet report, sending...
I20260926 02:52:56.930747 22675 ts_manager.cc:194] Registered new tserver with Master: 739d34d289f54e52aebca0738d3106a9 (127.19.185.194:38085)
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
I20260926 02:52:56.931643 22675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:54859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:52:56.957407 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34979
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:57.289204 23003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:57.289544 23003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:57.289717 23003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:57.296841 23003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:57.297084 23003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:57.297281 23003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:52:57.307106 23003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:57.308491 23003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:57.309719 23003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:57.322568 23008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:57.322669 23011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:57.322669 23009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:57.323678 23003 server_base.cc:1034] running on GCE node
I20260926 02:52:57.324265 23003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:57.325145 23003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:57.326376 23003 hybrid_clock.cc:648] HybridClock initialized: now 1790391177326313 us; error 39 us; skew 500 ppm
I20260926 02:52:57.326678 23003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:57 dist-test-slave-ktng krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391177, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:57.330113 23003 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:52:57.331682 23003 webserver.cc:460] Webserver started at http://127.19.185.195:45419/ using document root <none> and password file <none>
I20260926 02:52:57.332378 23003 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:57.332508 23003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:57.332890 23003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:57.336018 23003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "83a487b1d8ac417291a9da816c6a26d2"
format_stamp: "Formatted at 2026-09-26 02:52:57 on dist-test-slave-ktng"
server_key: "9c55dab224cfa88aea8fbd2c9f5abbc7"
server_key_iv: "749a76007de6d1adae48ae9842211343"
server_key_version: "encryptionkey@0"
I20260926 02:52:57.337369 23003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "83a487b1d8ac417291a9da816c6a26d2"
format_stamp: "Formatted at 2026-09-26 02:52:57 on dist-test-slave-ktng"
server_key: "9c55dab224cfa88aea8fbd2c9f5abbc7"
server_key_iv: "749a76007de6d1adae48ae9842211343"
server_key_version: "encryptionkey@0"
I20260926 02:52:57.343094 23003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 02:52:57.346771 23018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:57.348340 23003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:52:57.348527 23003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "83a487b1d8ac417291a9da816c6a26d2"
format_stamp: "Formatted at 2026-09-26 02:52:57 on dist-test-slave-ktng"
server_key: "9c55dab224cfa88aea8fbd2c9f5abbc7"
server_key_iv: "749a76007de6d1adae48ae9842211343"
server_key_version: "encryptionkey@0"
I20260926 02:52:57.348744 23003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:57.370290 23003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:57.374589 23003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:57.374891 23003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:57.375939 23003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:52:57.377211 23003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:52:57.377315 23003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:57.377424 23003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:52:57.377468 23003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:57.411896 23003 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:41089
I20260926 02:52:57.411993 23130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:41089 every 8 connection(s)
I20260926 02:52:57.413265 23003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:52:57.416153 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 23003
I20260926 02:52:57.416338 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:52:57.416792 20199 external_mini_cluster.cc:1428] Setting key b67ff0980ee582a0c0a59706b57091ed
I20260926 02:52:57.417094 22864 heartbeater.cc:503] Master 127.19.185.254:34979 was elected leader, sending a full tablet report...
Sep 26 02:52:57 dist-test-slave-ktng krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391177, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:57.435838 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:57.415988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:48981 (local address 127.19.185.254:34979)
0926 02:52:57.416220 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:57.416229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:57.417687 (+ 1458us) server_negotiation.cc:407] Connection header received
0926 02:52:57.420078 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:57.420082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:57.420140 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:57.420212 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:57.421650 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:57.422541 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:57.424441 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:57.424912 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:57.425026 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:57.429147 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:57.429152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:57.429156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:57.429192 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:57.432465 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:57.433341 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:57.433353 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:57.433356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:57.433481 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:57.433946 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:57.433953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:57.433956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:57.434309 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:57.434477 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:57.435223 (+ 746us) server_negotiation.cc:299] Negotiation successful
0926 02:52:57.435466 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:52:57.436829 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:57.416538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:34979 (local address 127.19.185.195:48981)
0926 02:52:57.417518 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0926 02:52:57.417562 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:52:57.419810 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:52:57.420428 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:52:57.420435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:52:57.420928 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:52:57.421489 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:57.421506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:57.422744 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:57.422749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:57.424296 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:52:57.424309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:52:57.425196 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:52:57.425202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:52:57.425404 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:57.426300 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:52:57.426329 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:52:57.428828 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:52:57.432638 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:57.432646 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:57.432663 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:57.433210 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:57.433618 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:52:57.433622 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:52:57.433624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:52:57.433814 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:52:57.434453 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:52:57.434462 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:52:57.434829 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 02:52:57.435102 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 02:52:57.435692 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":462,"threads_started":1}
I20260926 02:52:57.438474 23131 heartbeater.cc:347] Connected to a master server at 127.19.185.254:34979
I20260926 02:52:57.438756 23131 heartbeater.cc:464] Registering TS with master...
I20260926 02:52:57.439667 23131 heartbeater.cc:511] Master 127.19.185.254:34979 requested a full tablet report, sending...
I20260926 02:52:57.441251 22675 ts_manager.cc:194] Registered new tserver with Master: 83a487b1d8ac417291a9da816c6a26d2 (127.19.185.195:41089)
I20260926 02:52:57.442077 22675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:48981
I20260926 02:52:57.445645 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 02:52:57.477511 22866 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:52:57.474775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.19.185.254:34979)
0926 02:52:57.474982 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:57.474989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:57.475527 (+ 538us) server_negotiation.cc:407] Connection header received
0926 02:52:57.475667 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:57.475671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:57.475736 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:57.475879 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:57.477185 (+ 1306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55326: BlockingRecv error: recv got EOF from 127.0.0.1:55326 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:52:57.480814 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 22736
W20260926 02:52:57.483860 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 22736: Not authorized: Client connection negotiation failed: client connection to 127.19.185.193:46579: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:52:57.484057 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 22736
I20260926 02:52:57.502275 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 22870
W20260926 02:52:57.505220 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 22870: Not authorized: Client connection negotiation failed: client connection to 127.19.185.194:38085: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:52:57.505409 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 22870
I20260926 02:52:57.519641 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 23003
W20260926 02:52:57.522692 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 23003: Not authorized: Client connection negotiation failed: client connection to 127.19.185.195:41089: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:52:57.522832 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 23003
I20260926 02:52:57.537995 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 22643
I20260926 02:52:57.539247 22705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:52:57.642498 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 22643
2026-09-26T02:52:57Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3877 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:52:57 dist-test-slave-ktng krb5kdc[23148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:52:57 dist-test-slave-ktng krb5kdc[23148](info): set up 2 sockets
Sep 26 02:52:57 dist-test-slave-ktng krb5kdc[23148](info): commencing operation
krb5kdc: starting...
W20260926 02:52:58.780738 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:52:58 dist-test-slave-ktng krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:52:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:52:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:52:58.974690 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:34495
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:34495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:52:59.296663 23161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:52:59.297031 23161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:52:59.297204 23161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:52:59.304404 23161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:52:59.304572 23161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:52:59.304684 23161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:52:59.304790 23161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:52:59.304883 23161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:52:59.314590 23161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:34495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:34495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:52:59.316013 23161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:52:59.317358 23161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:52:59.328966 23167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:59.329254 23166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:52:59.329169 23169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:52:59.330313 23161 server_base.cc:1034] running on GCE node
I20260926 02:52:59.331230 23161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:52:59.332604 23161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:52:59.333832 23161 hybrid_clock.cc:648] HybridClock initialized: now 1790391179333791 us; error 34 us; skew 500 ppm
I20260926 02:52:59.334196 23161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:52:59 dist-test-slave-ktng krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391179, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:52:59.337756 23161 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:52:59.339229 23161 webserver.cc:460] Webserver started at http://127.19.185.254:46507/ using document root <none> and password file <none>
I20260926 02:52:59.339948 23161 fs_manager.cc:359] Metadata directory not provided
I20260926 02:52:59.340098 23161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:52:59.340402 23161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:52:59.343331 23161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "3130cf45bb5b41b89854a0a89cc789a1"
format_stamp: "Formatted at 2026-09-26 02:52:59 on dist-test-slave-ktng"
server_key: "c0562634be16f4879b16dfe50b164526"
server_key_iv: "56034eceaa382f3952a3b60cab084c20"
server_key_version: "encryptionkey@0"
I20260926 02:52:59.344118 23161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "3130cf45bb5b41b89854a0a89cc789a1"
format_stamp: "Formatted at 2026-09-26 02:52:59 on dist-test-slave-ktng"
server_key: "c0562634be16f4879b16dfe50b164526"
server_key_iv: "56034eceaa382f3952a3b60cab084c20"
server_key_version: "encryptionkey@0"
I20260926 02:52:59.349092 23161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 02:52:59.352334 23176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:52:59.353711 23161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:52:59.353911 23161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "3130cf45bb5b41b89854a0a89cc789a1"
format_stamp: "Formatted at 2026-09-26 02:52:59 on dist-test-slave-ktng"
server_key: "c0562634be16f4879b16dfe50b164526"
server_key_iv: "56034eceaa382f3952a3b60cab084c20"
server_key_version: "encryptionkey@0"
I20260926 02:52:59.354108 23161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:52:59.373610 23161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:52:59.377971 23161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:52:59.378268 23161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:52:59.396468 23161 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:34495
I20260926 02:52:59.396508 23227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:34495 every 8 connection(s)
I20260926 02:52:59.397821 23161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:52:59.401572 23228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:52:59.402630 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 23161
I20260926 02:52:59.402849 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:52:59.403693 20199 external_mini_cluster.cc:1428] Setting key ea7c0c1e943cdeadb13cf5cf213c6f0c
I20260926 02:52:59.416054 23228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1: Bootstrap starting.
Sep 26 02:52:59 dist-test-slave-ktng krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:52:59.419037 23228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1: Neither blocks nor log segments found. Creating new log.
I20260926 02:52:59.420375 23228 log.cc:824] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1: Log is configured to *not* fsync() on all Append() calls
I20260926 02:52:59.423854 23228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1: No bootstrap required, opened a new log
I20260926 02:52:59.428305 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:59.405663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46512 (local address 127.19.185.254:34495)
0926 02:52:59.406419 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:59.406434 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:59.406484 (+ 50us) server_negotiation.cc:407] Connection header received
0926 02:52:59.408557 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:59.408584 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:59.409090 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:59.409522 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:59.410913 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:59.412931 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:59.414228 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:59.415236 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:59.415434 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:59.419445 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:59.419459 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:59.419481 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:52:59.419541 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:59.423588 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:59.424361 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:59.424369 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:59.424376 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:59.424590 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:52:59.424928 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:52:59.424932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:52:59.424936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:52:59.425620 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:59.425907 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:59.426326 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 02:52:59.426776 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":301,"threads_started":1}
I20260926 02:52:59.432391 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:52:59.433666 23228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3130cf45bb5b41b89854a0a89cc789a1" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34495 } }
I20260926 02:52:59.434015 23228 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:52:59.434113 23228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3130cf45bb5b41b89854a0a89cc789a1, State: Initialized, Role: FOLLOWER
I20260926 02:52:59.434687 23228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3130cf45bb5b41b89854a0a89cc789a1" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34495 } }
I20260926 02:52:59.434899 23228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:52:59.435025 23228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:52:59.435190 23228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:52:59.437342 23228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3130cf45bb5b41b89854a0a89cc789a1" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34495 } }
I20260926 02:52:59.437836 23228 leader_election.cc:302] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3130cf45bb5b41b89854a0a89cc789a1; no voters:
I20260926 02:52:59.438472 23228 leader_election.cc:288] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:52:59.438593 23233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:52:59.439996 23233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3130cf45bb5b41b89854a0a89cc789a1, State: Running, Role: LEADER
I20260926 02:52:59.440490 23233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3130cf45bb5b41b89854a0a89cc789a1" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34495 } }
I20260926 02:52:59.440809 23228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:52:59.443209 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3130cf45bb5b41b89854a0a89cc789a1. Latest consensus state: current_term: 1 leader_uuid: "3130cf45bb5b41b89854a0a89cc789a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3130cf45bb5b41b89854a0a89cc789a1" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34495 } } }
I20260926 02:52:59.443174 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3130cf45bb5b41b89854a0a89cc789a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3130cf45bb5b41b89854a0a89cc789a1" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34495 } } }
I20260926 02:52:59.443642 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [sys.catalog]: This master's current role is: LEADER
I20260926 02:52:59.443722 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1 [sys.catalog]: This master's current role is: LEADER
I20260926 02:52:59.454164 23240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:52:59.460167 23240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:52:59.466872 23252 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:52:59.470115 23240 catalog_manager.cc:1347] Generated new cluster ID: b555653bf5714b778382f9553c838a33
I20260926 02:52:59.470229 23240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:52:59.483875 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:52:59.476405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.19.185.254:34495)
0926 02:52:59.476583 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:52:59.476590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:52:59.476760 (+ 170us) server_negotiation.cc:407] Connection header received
0926 02:52:59.477036 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:52:59.477040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:52:59.477105 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:52:59.477218 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:52:59.478255 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:59.478963 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:59.480328 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:52:59.480777 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:52:59.480917 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:52:59.482257 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:52:59.482262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:52:59.482265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:52:59.482940 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:52:59.483011 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:52:59.483039 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:52:59.483166 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 02:52:59.483329 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 02:52:59.483422 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:52:59.500200 23240 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:52:59.501572 23240 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:52:59.514428 23240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3130cf45bb5b41b89854a0a89cc789a1: Generated new TSK 0
I20260926 02:52:59.515410 23240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:52:59.527446 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 23161
I20260926 02:52:59.528672 23223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:52:59.623122 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 23161
2026-09-26T02:52:59Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1975 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:52:59 dist-test-slave-ktng krb5kdc[23262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:52:59 dist-test-slave-ktng krb5kdc[23262](info): set up 2 sockets
Sep 26 02:52:59 dist-test-slave-ktng krb5kdc[23262](info): commencing operation
krb5kdc: starting...
W20260926 02:53:00.704699 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:53:00 dist-test-slave-ktng krb5kdc[23262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:53:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:53:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:53:00.902433 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:38087
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:38087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:01.224800 23275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:01.225143 23275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:01.225306 23275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:01.232368 23275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:01.232507 23275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:01.232600 23275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:01.232707 23275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:01.232791 23275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:53:01.242599 23275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:38087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:38087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:01.243991 23275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:01.245258 23275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:01.256850 23280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:01.256878 23283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:01.256809 23281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:01.257682 23275 server_base.cc:1034] running on GCE node
I20260926 02:53:01.258476 23275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:01.259976 23275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:01.261219 23275 hybrid_clock.cc:648] HybridClock initialized: now 1790391181261153 us; error 52 us; skew 500 ppm
I20260926 02:53:01.261552 23275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:01 dist-test-slave-ktng krb5kdc[23262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391181, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:01.264628 23275 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:53:01.266013 23275 webserver.cc:460] Webserver started at http://127.19.185.254:37839/ using document root <none> and password file <none>
I20260926 02:53:01.266613 23275 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:01.266718 23275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:01.266964 23275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:01.269852 23275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "ed653d18059a4875b8a3757e3fa64757"
format_stamp: "Formatted at 2026-09-26 02:53:01 on dist-test-slave-ktng"
server_key: "ee0efe9b2680d35c544ae689cea88e79"
server_key_iv: "117f0a679f5185563b9530c18d5b962c"
server_key_version: "encryptionkey@0"
I20260926 02:53:01.270526 23275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "ed653d18059a4875b8a3757e3fa64757"
format_stamp: "Formatted at 2026-09-26 02:53:01 on dist-test-slave-ktng"
server_key: "ee0efe9b2680d35c544ae689cea88e79"
server_key_iv: "117f0a679f5185563b9530c18d5b962c"
server_key_version: "encryptionkey@0"
I20260926 02:53:01.275484 23275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:53:01.278770 23290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:01.280613 23275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:53:01.280819 23275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "ed653d18059a4875b8a3757e3fa64757"
format_stamp: "Formatted at 2026-09-26 02:53:01 on dist-test-slave-ktng"
server_key: "ee0efe9b2680d35c544ae689cea88e79"
server_key_iv: "117f0a679f5185563b9530c18d5b962c"
server_key_version: "encryptionkey@0"
I20260926 02:53:01.281000 23275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:01.297746 23275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:01.301793 23275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:01.302084 23275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:01.320345 23275 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:38087
I20260926 02:53:01.320369 23341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:38087 every 8 connection(s)
I20260926 02:53:01.321689 23275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:53:01.325284 23342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:01.330415 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 23275
I20260926 02:53:01.330613 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:53:01.331379 20199 external_mini_cluster.cc:1428] Setting key c424d4b10caaf9767e60cca3e482a453
I20260926 02:53:01.336503 23342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757: Bootstrap starting.
I20260926 02:53:01.339493 23342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:01.340461 23342 log.cc:824] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:01.343125 23342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757: No bootstrap required, opened a new log
I20260926 02:53:01.349879 23342 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed653d18059a4875b8a3757e3fa64757" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38087 } }
I20260926 02:53:01.350235 23342 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:01.350339 23342 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed653d18059a4875b8a3757e3fa64757, State: Initialized, Role: FOLLOWER
Sep 26 02:53:01 dist-test-slave-ktng krb5kdc[23262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:01.350975 23342 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed653d18059a4875b8a3757e3fa64757" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38087 } }
I20260926 02:53:01.351209 23342 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:53:01.351373 23342 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:53:01.351550 23342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:01.354275 23342 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed653d18059a4875b8a3757e3fa64757" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38087 } }
I20260926 02:53:01.354779 23342 leader_election.cc:302] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed653d18059a4875b8a3757e3fa64757; no voters:
I20260926 02:53:01.355459 23342 leader_election.cc:288] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:53:01.355572 23347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:01.357036 23347 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [term 1 LEADER]: Becoming Leader. State: Replica: ed653d18059a4875b8a3757e3fa64757, State: Running, Role: LEADER
I20260926 02:53:01.357625 23347 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed653d18059a4875b8a3757e3fa64757" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38087 } }
I20260926 02:53:01.357828 23342 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:01.361043 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:01.333452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.19.185.254:38087)
0926 02:53:01.334236 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:01.334250 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:01.334293 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:53:01.336007 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:01.336029 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:01.336398 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:01.336817 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:01.338219 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:01.339613 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:01.340902 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:01.341606 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:01.341798 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:01.351898 (+ 10100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:01.351906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:01.351921 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:01.351961 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:01.355104 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:01.356094 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:01.356105 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:01.356121 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:01.356768 (+ 647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:01.357314 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:01.357321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:01.357325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:01.358120 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:01.358426 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:01.358840 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 02:53:01.359333 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":366,"threads_started":1}
I20260926 02:53:01.363548 23348 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed653d18059a4875b8a3757e3fa64757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed653d18059a4875b8a3757e3fa64757" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38087 } } }
I20260926 02:53:01.363996 23348 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:01.367635 23349 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed653d18059a4875b8a3757e3fa64757. Latest consensus state: current_term: 1 leader_uuid: "ed653d18059a4875b8a3757e3fa64757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed653d18059a4875b8a3757e3fa64757" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38087 } } }
I20260926 02:53:01.368018 23349 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:01.370627 23354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:53:01.375771 23354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:53:01.385304 23354 catalog_manager.cc:1347] Generated new cluster ID: 372066ee742e4f339e6ba0d40d60b905
I20260926 02:53:01.385612 23354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:53:01.395478 23354 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:53:01.397281 23354 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:53:01.409040 23354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed653d18059a4875b8a3757e3fa64757: Generated new TSK 0
I20260926 02:53:01.410444 23354 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:53:01.466432 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38087
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:01.788552 23368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:01.788872 23368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:01.789034 23368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:01.796154 23368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:01.796298 23368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:01.796473 23368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:53:01.806142 23368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:01.807473 23368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:01.808823 23368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:01.821941 23373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:01.821878 23376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:01.821920 23374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:01.822878 23368 server_base.cc:1034] running on GCE node
I20260926 02:53:01.823571 23368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:01.824501 23368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:01.825699 23368 hybrid_clock.cc:648] HybridClock initialized: now 1790391181825671 us; error 36 us; skew 500 ppm
I20260926 02:53:01.826009 23368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:01 dist-test-slave-ktng krb5kdc[23262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391181, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:01.829238 23368 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:53:01.830678 23368 webserver.cc:460] Webserver started at http://127.19.185.193:43647/ using document root <none> and password file <none>
I20260926 02:53:01.831408 23368 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:01.831538 23368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:01.831858 23368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:01.834746 23368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "0e932c194fb44f0f9b9a21779fcdabe0"
format_stamp: "Formatted at 2026-09-26 02:53:01 on dist-test-slave-ktng"
server_key: "3da18d324c16adcc095ea930e702d17d"
server_key_iv: "d1aa3e6eeedaaec9746fc63245d7f14b"
server_key_version: "encryptionkey@0"
I20260926 02:53:01.835590 23368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "0e932c194fb44f0f9b9a21779fcdabe0"
format_stamp: "Formatted at 2026-09-26 02:53:01 on dist-test-slave-ktng"
server_key: "3da18d324c16adcc095ea930e702d17d"
server_key_iv: "d1aa3e6eeedaaec9746fc63245d7f14b"
server_key_version: "encryptionkey@0"
I20260926 02:53:01.840583 23368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 02:53:01.844015 23383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:01.845368 23368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:53:01.845533 23368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "0e932c194fb44f0f9b9a21779fcdabe0"
format_stamp: "Formatted at 2026-09-26 02:53:01 on dist-test-slave-ktng"
server_key: "3da18d324c16adcc095ea930e702d17d"
server_key_iv: "d1aa3e6eeedaaec9746fc63245d7f14b"
server_key_version: "encryptionkey@0"
I20260926 02:53:01.845731 23368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:01.877852 23368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:01.882458 23368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:01.882763 23368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:01.883826 23368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:01.885263 23368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:01.885365 23368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:01.885519 23368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:01.885599 23368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:01.918954 23368 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:40049
I20260926 02:53:01.918987 23497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:40049 every 8 connection(s)
I20260926 02:53:01.920290 23368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:53:01.926415 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 23368
I20260926 02:53:01.926620 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:53:01.927448 20199 external_mini_cluster.cc:1428] Setting key 178ba718663c87e62374831acd28fb57
Sep 26 02:53:01 dist-test-slave-ktng krb5kdc[23262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391181, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:01.940152 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:01.922862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:38709 (local address 127.19.185.254:38087)
0926 02:53:01.923395 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:01.923402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:01.924293 (+ 891us) server_negotiation.cc:407] Connection header received
0926 02:53:01.926240 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:01.926244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:01.926317 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:01.926411 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:01.928366 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:01.929124 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:01.930649 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:01.931026 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:01.931160 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:01.934770 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:01.934774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:01.934778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:01.934807 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:01.937193 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:01.937866 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:01.937877 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:01.937880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:01.937996 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:01.938448 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:01.938453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:01.938456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:01.938705 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:01.938849 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:01.939566 (+ 717us) server_negotiation.cc:299] Negotiation successful
0926 02:53:01.939787 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":274,"threads_started":1}
I20260926 02:53:01.941170 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:01.923439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38087 (local address 127.19.185.193:38709)
0926 02:53:01.924132 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:01.924181 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:01.925786 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:01.926697 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:01.926710 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:01.927463 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:01.928200 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:01.928220 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:01.929276 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:01.929288 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:01.930490 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:01.930501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:01.931254 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:01.931261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:01.931507 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:01.932202 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:01.932223 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:01.934555 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:01.937339 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:01.937348 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:01.937363 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:01.937745 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:01.938109 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:01.938114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:01.938117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:01.938331 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:01.938838 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:01.938845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:01.939161 (+ 316us) client_negotiation.cc:769] Sending connection context
0926 02:53:01.939459 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 02:53:01.939965 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":272,"threads_started":1}
I20260926 02:53:01.942875 23498 heartbeater.cc:347] Connected to a master server at 127.19.185.254:38087
I20260926 02:53:01.943161 23498 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:01.943903 23498 heartbeater.cc:511] Master 127.19.185.254:38087 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
I20260926 02:53:01.945993 23307 ts_manager.cc:194] Registered new tserver with Master: 0e932c194fb44f0f9b9a21779fcdabe0 (127.19.185.193:40049)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
I20260926 02:53:01.947674 23307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:38709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:53:01.971463 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38087
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:02.292842 23504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:02.293136 23504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:02.293233 23504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:02.300591 23504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:02.300705 23504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:02.300817 23504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:53:02.310572 23504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:02.311868 23504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:02.313102 23504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:02.326614 23509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:02.326581 23510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:02.327430 23512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:02.327888 23504 server_base.cc:1034] running on GCE node
I20260926 02:53:02.328583 23504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:02.329644 23504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:02.330884 23504 hybrid_clock.cc:648] HybridClock initialized: now 1790391182330818 us; error 59 us; skew 500 ppm
I20260926 02:53:02.331250 23504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:02 dist-test-slave-ktng krb5kdc[23262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391182, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:02.334995 23504 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:53:02.338907 23504 webserver.cc:460] Webserver started at http://127.19.185.194:36285/ using document root <none> and password file <none>
I20260926 02:53:02.339663 23504 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:02.339789 23504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:02.340096 23504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:02.342973 23504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "921512f06a66495d8221857dc04e3582"
format_stamp: "Formatted at 2026-09-26 02:53:02 on dist-test-slave-ktng"
server_key: "d7a29e31694c8b50e85a5cec73284478"
server_key_iv: "9fce0c4e6ba7474ad0c5a9ce7ae41931"
server_key_version: "encryptionkey@0"
I20260926 02:53:02.343817 23504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "921512f06a66495d8221857dc04e3582"
format_stamp: "Formatted at 2026-09-26 02:53:02 on dist-test-slave-ktng"
server_key: "d7a29e31694c8b50e85a5cec73284478"
server_key_iv: "9fce0c4e6ba7474ad0c5a9ce7ae41931"
server_key_version: "encryptionkey@0"
I20260926 02:53:02.348860 23504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:53:02.352288 23519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:02.353634 23504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:53:02.353894 23504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "921512f06a66495d8221857dc04e3582"
format_stamp: "Formatted at 2026-09-26 02:53:02 on dist-test-slave-ktng"
server_key: "d7a29e31694c8b50e85a5cec73284478"
server_key_iv: "9fce0c4e6ba7474ad0c5a9ce7ae41931"
server_key_version: "encryptionkey@0"
I20260926 02:53:02.354074 23504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:02.370059 23504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:02.374230 23504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:02.374512 23504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:02.375501 23504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:02.376820 23504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:02.376890 23504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:02.377045 23504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:02.377108 23504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:02.410427 23504 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:39549
I20260926 02:53:02.410458 23632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:39549 every 8 connection(s)
I20260926 02:53:02.411775 23504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:53:02.419921 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 23504
I20260926 02:53:02.420105 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:53:02.420542 20199 external_mini_cluster.cc:1428] Setting key fd88b41b4366a17ac27076c659026e52
Sep 26 02:53:02 dist-test-slave-ktng krb5kdc[23262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391182, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:02.433005 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:02.414343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:34667 (local address 127.19.185.254:38087)
0926 02:53:02.414548 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:02.414555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:02.415720 (+ 1165us) server_negotiation.cc:407] Connection header received
0926 02:53:02.417418 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:02.417422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:02.417489 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:02.417576 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:02.419010 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:02.419938 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:02.421876 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:02.422524 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:02.422652 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:02.426240 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:02.426249 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:02.426257 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:02.426304 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:02.429747 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:02.430463 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:02.430472 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:02.430479 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:02.430637 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:02.431026 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:02.431034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:02.431040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:02.431462 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:02.431615 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:02.432299 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 02:53:02.432551 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:53:02.434041 23635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:02.414774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38087 (local address 127.19.185.194:34667)
0926 02:53:02.415543 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:02.415585 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:02.417139 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:02.417780 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:02.417788 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:02.418315 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:02.418863 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:02.418879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:02.420135 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:02.420157 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:02.421705 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:02.421719 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:02.422733 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:02.422739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:02.422923 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:02.423641 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:02.423666 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:02.425992 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:02.429887 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:02.429894 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:02.429907 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:02.430352 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:02.430748 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:02.430751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:02.430754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:02.430920 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:02.431607 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:02.431613 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:02.431924 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 02:53:02.432168 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 02:53:02.432555 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":286,"threads_started":1}
I20260926 02:53:02.435654 23633 heartbeater.cc:347] Connected to a master server at 127.19.185.254:38087
I20260926 02:53:02.435966 23633 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:02.436774 23633 heartbeater.cc:511] Master 127.19.185.254:38087 requested a full tablet report, sending...
I20260926 02:53:02.438279 23307 ts_manager.cc:194] Registered new tserver with Master: 921512f06a66495d8221857dc04e3582 (127.19.185.194:39549)
I20260926 02:53:02.439538 23307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:34667
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:53:02.466718 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38087
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:02.789544 23638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:02.789872 23638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:02.790066 23638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:02.797650 23638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:02.797806 23638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:02.798013 23638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:53:02.807934 23638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:02.809321 23638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:02.810662 23638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:02.823180 23643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:02.824630 23644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:02.824659 23646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:02.825515 23638 server_base.cc:1034] running on GCE node
I20260926 02:53:02.826102 23638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:02.827003 23638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:02.828266 23638 hybrid_clock.cc:648] HybridClock initialized: now 1790391182828215 us; error 59 us; skew 500 ppm
I20260926 02:53:02.828655 23638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:02 dist-test-slave-ktng krb5kdc[23262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391182, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:02.831951 23638 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:53:02.833334 23638 webserver.cc:460] Webserver started at http://127.19.185.195:39971/ using document root <none> and password file <none>
I20260926 02:53:02.834009 23638 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:02.834141 23638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:02.834451 23638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:02.837320 23638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "1ea2a811a1e1422cbe86f66dbe19d378"
format_stamp: "Formatted at 2026-09-26 02:53:02 on dist-test-slave-ktng"
server_key: "88e8b0167bf427155b3c66b7c01b4dd6"
server_key_iv: "b8f80b84e4721e194361f888ee64bf7e"
server_key_version: "encryptionkey@0"
I20260926 02:53:02.838073 23638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "1ea2a811a1e1422cbe86f66dbe19d378"
format_stamp: "Formatted at 2026-09-26 02:53:02 on dist-test-slave-ktng"
server_key: "88e8b0167bf427155b3c66b7c01b4dd6"
server_key_iv: "b8f80b84e4721e194361f888ee64bf7e"
server_key_version: "encryptionkey@0"
I20260926 02:53:02.842891 23638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 02:53:02.846206 23653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:02.847558 23638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:53:02.847715 23638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "1ea2a811a1e1422cbe86f66dbe19d378"
format_stamp: "Formatted at 2026-09-26 02:53:02 on dist-test-slave-ktng"
server_key: "88e8b0167bf427155b3c66b7c01b4dd6"
server_key_iv: "b8f80b84e4721e194361f888ee64bf7e"
server_key_version: "encryptionkey@0"
I20260926 02:53:02.847891 23638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:02.874914 23638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:02.879431 23638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:02.879738 23638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:02.880774 23638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:02.882187 23638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:02.882257 23638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:02.882386 23638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:02.882434 23638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:02.916309 23638 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:45571
I20260926 02:53:02.916352 23766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:45571 every 8 connection(s)
I20260926 02:53:02.917634 23638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:53:02.925897 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 23638
I20260926 02:53:02.926095 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:53:02.926636 20199 external_mini_cluster.cc:1428] Setting key a2c29a3c51de0d3f71164c9dea3167fc
Sep 26 02:53:02 dist-test-slave-ktng krb5kdc[23262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391182, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:02.938668 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:02.920321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:59553 (local address 127.19.185.254:38087)
0926 02:53:02.920544 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:02.920552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:02.922071 (+ 1519us) server_negotiation.cc:407] Connection header received
0926 02:53:02.924550 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:02.924555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:02.924635 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:02.924745 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:02.926488 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:02.927421 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:02.929423 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:02.929942 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:02.930039 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:02.933466 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:02.933470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:02.933474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:02.933504 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:02.935863 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:02.936640 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:02.936644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:02.936647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:02.936763 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:02.937216 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:02.937219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:02.937221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:02.937460 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:02.937565 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:02.938248 (+ 683us) server_negotiation.cc:299] Negotiation successful
0926 02:53:02.938398 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:53:02.939706 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:02.920800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38087 (local address 127.19.185.195:59553)
0926 02:53:02.921888 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:02.921938 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:02.924243 (+ 2305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:02.924971 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:02.924979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:02.925503 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:02.926311 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:02.926337 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:02.927553 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:02.927558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:02.929283 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:02.929295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:02.930148 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:02.930153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:02.930395 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:02.931140 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:02.931162 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:02.933287 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:02.936005 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:02.936014 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:02.936032 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:02.936521 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:02.936894 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:02.936898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:02.936901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:02.937107 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:02.937593 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:02.937599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:02.937915 (+ 316us) client_negotiation.cc:769] Sending connection context
0926 02:53:02.938165 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 02:53:02.938555 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":604,"threads_started":1}
I20260926 02:53:02.941241 23767 heartbeater.cc:347] Connected to a master server at 127.19.185.254:38087
I20260926 02:53:02.941560 23767 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:02.942386 23767 heartbeater.cc:511] Master 127.19.185.254:38087 requested a full tablet report, sending...
I20260926 02:53:02.943622 23306 ts_manager.cc:194] Registered new tserver with Master: 1ea2a811a1e1422cbe86f66dbe19d378 (127.19.185.195:45571)
I20260926 02:53:02.944423 23306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:59553
I20260926 02:53:02.951524 23498 heartbeater.cc:503] Master 127.19.185.254:38087 was elected leader, sending a full tablet report...
I20260926 02:53:02.954170 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:53:02 dist-test-slave-ktng krb5kdc[23262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391182, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 02:53:02 dist-test-slave-ktng krb5kdc[23262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391182, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:03.001559 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:02.988219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.19.185.254:38087)
0926 02:53:02.988471 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:02.988478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:02.988930 (+ 452us) server_negotiation.cc:407] Connection header received
0926 02:53:02.989079 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:02.989084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:02.989149 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:02.989239 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:02.990644 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:02.991496 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:02.992705 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:02.993139 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:02.993251 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:02.996893 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:02.996897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:02.996901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:02.996935 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:02.999267 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:02.999885 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:02.999889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:02.999891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:03.000021 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:03.000350 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:03.000354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:03.000356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:03.000676 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:03.000846 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:03.001072 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 02:53:03.001223 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 02:53:03.002095 23306 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50534
I20260926 02:53:03.003976 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 23368
Sep 26 02:53:03 dist-test-slave-ktng krb5kdc[23262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391182, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:53:03.019580 23778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:03.005015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.19.185.193:40049)
0926 02:53:03.005620 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:03.005627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:03.005652 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:53:03.005707 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:03.005713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:03.005936 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:03.006169 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:03.007291 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:03.008415 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:03.009594 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:03.010213 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:03.010310 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:03.014239 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:03.014250 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:03.014266 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:03.014302 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:03.016949 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:03.017547 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:03.017554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:03.017556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:03.017667 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:03.018081 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:03.018086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:03.018092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:03.018519 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:03.018762 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:03.019012 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 02:53:03.019223 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":318,"threads_started":1}
W20260926 02:53:03.020874 23493 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33022
W20260926 02:53:03.021952 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 23368: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:53:03.022167 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 23368
I20260926 02:53:03.038213 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 23504
Sep 26 02:53:03 dist-test-slave-ktng krb5kdc[23262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391182, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:53:03.053999 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:03.039089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.19.185.194:39549)
0926 02:53:03.039633 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:03.039642 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:03.039669 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:53:03.039731 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:03.039737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:03.039951 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:03.040081 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:03.041265 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:03.042244 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:03.043530 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:03.044074 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:03.044185 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:03.048396 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:03.048408 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:03.048423 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:03.048458 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:03.051370 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:03.052114 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:03.052121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:03.052124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:03.052269 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:03.052686 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:03.052689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:03.052691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:03.053014 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:03.053265 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:03.053517 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 02:53:03.053713 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":280,"threads_started":1}
W20260926 02:53:03.055400 23628 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38840
W20260926 02:53:03.056492 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 23504: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:53:03.056654 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 23504
I20260926 02:53:03.072649 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 23638
Sep 26 02:53:03 dist-test-slave-ktng krb5kdc[23262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391182, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:53:03.089903 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:03.073733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45252 (local address 127.19.185.195:45571)
0926 02:53:03.074360 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:03.074367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:03.074395 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:53:03.074453 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:03.074459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:03.074678 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:03.074857 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:03.076347 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:03.077399 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:03.078733 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:03.079378 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:03.079512 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:03.083967 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:03.083980 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:03.083995 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:03.084028 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:03.086927 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:03.087692 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:03.087699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:03.087701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:03.087826 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:03.088133 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:03.088136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:03.088138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:03.088491 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:03.088806 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:03.089311 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 02:53:03.089513 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":330,"threads_started":1}
W20260926 02:53:03.091419 23762 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45252
W20260926 02:53:03.092598 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 23638: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:53:03.092805 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 23638
I20260926 02:53:03.108348 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 23275
I20260926 02:53:03.109481 23337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:03.214828 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 23275
2026-09-26T02:53:03Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3596 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:53:03 dist-test-slave-ktng krb5kdc[23784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:53:03 dist-test-slave-ktng krb5kdc[23784](info): set up 2 sockets
Sep 26 02:53:03 dist-test-slave-ktng krb5kdc[23784](info): commencing operation
krb5kdc: starting...
W20260926 02:53:04.308573 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:53:04 dist-test-slave-ktng krb5kdc[23784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:53:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:53:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:53:04.505278 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:39473
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:39473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:04.835126 23797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:04.835498 23797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:04.835661 23797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:04.842994 23797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:04.843146 23797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:04.843258 23797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:04.843385 23797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:04.843483 23797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:53:04.853163 23797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:39473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:39473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:04.854478 23797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:04.855888 23797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:04.867623 23802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:04.867748 23803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:04.867880 23805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:04.868606 23797 server_base.cc:1034] running on GCE node
I20260926 02:53:04.869352 23797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:04.870664 23797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:04.871892 23797 hybrid_clock.cc:648] HybridClock initialized: now 1790391184871842 us; error 43 us; skew 500 ppm
I20260926 02:53:04.872335 23797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:04 dist-test-slave-ktng krb5kdc[23784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391184, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:04.875174 23797 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:53:04.876657 23797 webserver.cc:460] Webserver started at http://127.19.185.254:42235/ using document root <none> and password file <none>
I20260926 02:53:04.877327 23797 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:04.877466 23797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:04.877770 23797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:04.880750 23797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "f509dd8250ba4d56933159d2db6519a9"
format_stamp: "Formatted at 2026-09-26 02:53:04 on dist-test-slave-ktng"
server_key: "2c9d1401b520c093f2ca17be7d2c63e8"
server_key_iv: "47d0061fa31f3e72f14926436d25eb4a"
server_key_version: "encryptionkey@0"
I20260926 02:53:04.881516 23797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "f509dd8250ba4d56933159d2db6519a9"
format_stamp: "Formatted at 2026-09-26 02:53:04 on dist-test-slave-ktng"
server_key: "2c9d1401b520c093f2ca17be7d2c63e8"
server_key_iv: "47d0061fa31f3e72f14926436d25eb4a"
server_key_version: "encryptionkey@0"
I20260926 02:53:04.887069 23797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:53:04.890473 23812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:04.891851 23797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:53:04.892021 23797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "f509dd8250ba4d56933159d2db6519a9"
format_stamp: "Formatted at 2026-09-26 02:53:04 on dist-test-slave-ktng"
server_key: "2c9d1401b520c093f2ca17be7d2c63e8"
server_key_iv: "47d0061fa31f3e72f14926436d25eb4a"
server_key_version: "encryptionkey@0"
I20260926 02:53:04.892220 23797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:04.904059 23797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:04.907887 23797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:04.908177 23797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:04.926268 23797 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:39473
I20260926 02:53:04.926301 23863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:39473 every 8 connection(s)
I20260926 02:53:04.927698 23797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:53:04.931361 23864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:04.933478 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 23797
I20260926 02:53:04.933691 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:53:04.934307 20199 external_mini_cluster.cc:1428] Setting key 06b73e2b9f0aeab9d8e03d94570649c2
I20260926 02:53:04.943801 23864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9: Bootstrap starting.
I20260926 02:53:04.947134 23864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:04.948108 23864 log.cc:824] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:53:04 dist-test-slave-ktng krb5kdc[23784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:04.951048 23864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9: No bootstrap required, opened a new log
I20260926 02:53:04.958112 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:04.936382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.19.185.254:39473)
0926 02:53:04.937155 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:04.937171 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:04.937218 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:53:04.939546 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:04.939574 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:04.940072 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:04.940520 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:04.941825 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:04.943748 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:04.944945 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:04.946006 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:04.946187 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:04.950295 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:04.950302 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:04.950315 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:04.950346 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:04.953502 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:04.954285 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:04.954290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:04.954297 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:04.954454 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:04.954859 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:04.954862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:04.954864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:04.955451 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:04.955686 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:04.956398 (+ 712us) server_negotiation.cc:299] Negotiation successful
0926 02:53:04.956961 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":319,"threads_started":1}
I20260926 02:53:04.960072 23864 raft_consensus.cc:348] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f509dd8250ba4d56933159d2db6519a9" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39473 } }
I20260926 02:53:04.960438 23864 raft_consensus.cc:374] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:04.960562 23864 raft_consensus.cc:729] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f509dd8250ba4d56933159d2db6519a9, State: Initialized, Role: FOLLOWER
I20260926 02:53:04.961277 23864 consensus_queue.cc:260] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f509dd8250ba4d56933159d2db6519a9" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39473 } }
I20260926 02:53:04.961558 23864 raft_consensus.cc:388] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:53:04.961688 23864 raft_consensus.cc:482] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:53:04.961859 23864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:04.964056 23864 raft_consensus.cc:504] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f509dd8250ba4d56933159d2db6519a9" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39473 } }
I20260926 02:53:04.964555 23864 leader_election.cc:302] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f509dd8250ba4d56933159d2db6519a9; no voters:
I20260926 02:53:04.965106 23864 leader_election.cc:288] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:53:04.965253 23869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:04.965986 23869 raft_consensus.cc:686] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [term 1 LEADER]: Becoming Leader. State: Replica: f509dd8250ba4d56933159d2db6519a9, State: Running, Role: LEADER
I20260926 02:53:04.966580 23869 consensus_queue.cc:237] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f509dd8250ba4d56933159d2db6519a9" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39473 } }
I20260926 02:53:04.967432 23864 sys_catalog.cc:564] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:04.970151 23870 sys_catalog.cc:455] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f509dd8250ba4d56933159d2db6519a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f509dd8250ba4d56933159d2db6519a9" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39473 } } }
I20260926 02:53:04.970045 23871 sys_catalog.cc:455] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f509dd8250ba4d56933159d2db6519a9. Latest consensus state: current_term: 1 leader_uuid: "f509dd8250ba4d56933159d2db6519a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f509dd8250ba4d56933159d2db6519a9" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39473 } } }
I20260926 02:53:04.970595 23871 sys_catalog.cc:458] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:04.970599 23870 sys_catalog.cc:458] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:04.974622 23877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:53:04.980068 23877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:53:04.990309 23877 catalog_manager.cc:1347] Generated new cluster ID: c0cc22ad4b7645c8862f3f2be2236c59
I20260926 02:53:04.990435 23877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:53:05.019431 23877 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:53:05.021677 23877 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:53:05.040153 23877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f509dd8250ba4d56933159d2db6519a9: Generated new TSK 0
I20260926 02:53:05.041325 23877 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:53:05.103287 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39473
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:05.428581 23890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:05.428858 23890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:05.429042 23890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:05.436139 23890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:05.436240 23890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:05.436441 23890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:53:05.446005 23890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:05.447335 23890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:05.448637 23890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:05.462339 23895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:05.462304 23896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:05.462383 23898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:05.462893 23890 server_base.cc:1034] running on GCE node
I20260926 02:53:05.463573 23890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:05.464488 23890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:05.465709 23890 hybrid_clock.cc:648] HybridClock initialized: now 1790391185465666 us; error 39 us; skew 500 ppm
I20260926 02:53:05.466009 23890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:05 dist-test-slave-ktng krb5kdc[23784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391185, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:05.469120 23890 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:53:05.470470 23890 webserver.cc:460] Webserver started at http://127.19.185.193:43491/ using document root <none> and password file <none>
I20260926 02:53:05.471073 23890 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:05.471171 23890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:05.471494 23890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:05.474303 23890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "dafc0cdb4bf5492289a3bc7623c7d31f"
format_stamp: "Formatted at 2026-09-26 02:53:05 on dist-test-slave-ktng"
server_key: "477bcd4c39442b7d0bbea55b717571b7"
server_key_iv: "a51b01f868b742945c67e8e47f149454"
server_key_version: "encryptionkey@0"
I20260926 02:53:05.475090 23890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "dafc0cdb4bf5492289a3bc7623c7d31f"
format_stamp: "Formatted at 2026-09-26 02:53:05 on dist-test-slave-ktng"
server_key: "477bcd4c39442b7d0bbea55b717571b7"
server_key_iv: "a51b01f868b742945c67e8e47f149454"
server_key_version: "encryptionkey@0"
I20260926 02:53:05.480206 23890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 02:53:05.483683 23905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:05.485008 23890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:53:05.485175 23890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "dafc0cdb4bf5492289a3bc7623c7d31f"
format_stamp: "Formatted at 2026-09-26 02:53:05 on dist-test-slave-ktng"
server_key: "477bcd4c39442b7d0bbea55b717571b7"
server_key_iv: "a51b01f868b742945c67e8e47f149454"
server_key_version: "encryptionkey@0"
I20260926 02:53:05.485375 23890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:05.504315 23890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:05.508937 23890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:05.509262 23890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:05.510298 23890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:05.511785 23890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:05.511893 23890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:05.512043 23890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:05.512125 23890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:05.546357 23890 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:35383
I20260926 02:53:05.546412 24019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:35383 every 8 connection(s)
I20260926 02:53:05.547737 23890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:53:05.554040 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 23890
I20260926 02:53:05.554252 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:53:05.554883 20199 external_mini_cluster.cc:1428] Setting key 6d51e766136e015721948f715b5f5b9d
Sep 26 02:53:05 dist-test-slave-ktng krb5kdc[23784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391185, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:05.568488 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:05.550296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:45363 (local address 127.19.185.254:39473)
0926 02:53:05.550723 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:05.550729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:05.551759 (+ 1030us) server_negotiation.cc:407] Connection header received
0926 02:53:05.553528 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:05.553533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:05.553652 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:05.553785 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:05.555460 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:05.556383 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:05.558045 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:05.558791 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:05.558940 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:05.562511 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:05.562516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:05.562520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:05.562550 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:05.565061 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:05.565989 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:05.565993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:05.565996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:05.566105 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:05.566639 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:05.566643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:05.566646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:05.566896 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:05.567007 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:05.567940 (+ 933us) server_negotiation.cc:299] Negotiation successful
0926 02:53:05.568097 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":204,"threads_started":1}
I20260926 02:53:05.569895 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:05.550867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39473 (local address 127.19.185.193:45363)
0926 02:53:05.551607 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:05.551646 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:05.553281 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:05.553982 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:05.553990 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:05.554641 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:05.555247 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:05.555264 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:05.556558 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:05.556563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:05.557884 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:05.557895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:05.559003 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:05.559007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:05.559213 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:05.559927 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:05.559953 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:05.562301 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:05.565196 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:05.565207 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:05.565227 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:05.565863 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:05.566264 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:05.566271 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:05.566277 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:05.566524 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:05.567022 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:05.567032 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:05.567520 (+ 488us) client_negotiation.cc:769] Sending connection context
0926 02:53:05.567838 (+ 318us) client_negotiation.cc:240] Negotiation successful
0926 02:53:05.568412 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":256,"threads_started":1}
I20260926 02:53:05.571540 24020 heartbeater.cc:347] Connected to a master server at 127.19.185.254:39473
I20260926 02:53:05.571857 24020 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:05.572561 24020 heartbeater.cc:511] Master 127.19.185.254:39473 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
I20260926 02:53:05.574574 23829 ts_manager.cc:194] Registered new tserver with Master: dafc0cdb4bf5492289a3bc7623c7d31f (127.19.185.193:35383)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
I20260926 02:53:05.576356 23829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:45363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:53:05.602224 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39473
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:05.916886 24026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:05.917343 24026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:05.917541 24026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:05.924923 24026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:05.925096 24026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:05.925292 24026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:53:05.935192 24026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:05.936514 24026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:05.937878 24026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:05.951566 24032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:05.951557 24033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:05.951575 24035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:05.952260 24026 server_base.cc:1034] running on GCE node
I20260926 02:53:05.952872 24026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:05.953851 24026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:05.955123 24026 hybrid_clock.cc:648] HybridClock initialized: now 1790391185955078 us; error 49 us; skew 500 ppm
I20260926 02:53:05.955528 24026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:05 dist-test-slave-ktng krb5kdc[23784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391185, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:05.959019 24026 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:53:05.960649 24026 webserver.cc:460] Webserver started at http://127.19.185.194:38003/ using document root <none> and password file <none>
I20260926 02:53:05.961408 24026 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:05.961568 24026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:05.961889 24026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:05.964948 24026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "33598b18bc084cd1ad046a270a71c9f5"
format_stamp: "Formatted at 2026-09-26 02:53:05 on dist-test-slave-ktng"
server_key: "1c04a15050a38d51adfe1fd2c0261951"
server_key_iv: "3db3579fbe726080381e3986f7f04095"
server_key_version: "encryptionkey@0"
I20260926 02:53:05.965962 24026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "33598b18bc084cd1ad046a270a71c9f5"
format_stamp: "Formatted at 2026-09-26 02:53:05 on dist-test-slave-ktng"
server_key: "1c04a15050a38d51adfe1fd2c0261951"
server_key_iv: "3db3579fbe726080381e3986f7f04095"
server_key_version: "encryptionkey@0"
I20260926 02:53:05.971287 24026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:53:05.974872 24042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:05.976568 24026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:53:05.976740 24026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "33598b18bc084cd1ad046a270a71c9f5"
format_stamp: "Formatted at 2026-09-26 02:53:05 on dist-test-slave-ktng"
server_key: "1c04a15050a38d51adfe1fd2c0261951"
server_key_iv: "3db3579fbe726080381e3986f7f04095"
server_key_version: "encryptionkey@0"
I20260926 02:53:05.976950 24026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:05.993351 24026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:05.997568 24026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:05.997857 24026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:05.998853 24026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:06.000283 24026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:06.000386 24026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:06.000536 24026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:06.000626 24026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:06.034462 24026 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:38263
I20260926 02:53:06.034559 24154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:38263 every 8 connection(s)
I20260926 02:53:06.035778 24026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:53:06.040268 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 24026
I20260926 02:53:06.040462 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:53:06.040891 20199 external_mini_cluster.cc:1428] Setting key 362e8b7a7a89a77b87d435f8ea0c337b
Sep 26 02:53:06 dist-test-slave-ktng krb5kdc[23784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391185, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:06.059903 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:06.038488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:54655 (local address 127.19.185.254:39473)
0926 02:53:06.038687 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:06.038692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:06.040024 (+ 1332us) server_negotiation.cc:407] Connection header received
0926 02:53:06.042297 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:06.042301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:06.042364 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:06.042448 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:06.044262 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.045427 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.047112 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.047872 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.048017 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:06.052587 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:06.052592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:06.052595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:06.052630 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:06.056219 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:06.057273 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:06.057278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:06.057282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:06.057430 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:06.057916 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:06.057921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:06.057941 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:06.058271 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:06.058433 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:06.059333 (+ 900us) server_negotiation.cc:299] Negotiation successful
0926 02:53:06.059566 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:53:06.061244 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:06.038934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39473 (local address 127.19.185.194:54655)
0926 02:53:06.039856 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:06.039898 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:06.042007 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:06.042775 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:06.042788 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:06.043479 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:06.044055 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:06.044073 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.045640 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.045644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:06.046951 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:06.046964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.048145 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.048151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:06.048523 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:06.049371 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:06.049409 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:06.052339 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:06.056416 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:06.056425 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:06.056441 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:06.057109 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:06.057555 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:06.057593 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:06.057595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:06.057787 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:06.058416 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:06.058424 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:06.058923 (+ 499us) client_negotiation.cc:769] Sending connection context
0926 02:53:06.059208 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 02:53:06.059812 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":477,"threads_started":1}
I20260926 02:53:06.062915 24155 heartbeater.cc:347] Connected to a master server at 127.19.185.254:39473
I20260926 02:53:06.063235 24155 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
I20260926 02:53:06.064061 24155 heartbeater.cc:511] Master 127.19.185.254:39473 requested a full tablet report, sending...
I20260926 02:53:06.065615 23829 ts_manager.cc:194] Registered new tserver with Master: 33598b18bc084cd1ad046a270a71c9f5 (127.19.185.194:38263)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
I20260926 02:53:06.066862 23829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:54655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:53:06.092522 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39473
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:06.413164 24160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:06.413443 24160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:06.413602 24160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:06.420781 24160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:06.420885 24160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:06.421098 24160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:53:06.430498 24160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:06.431821 24160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:06.433143 24160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:06.445955 24165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:06.445987 24168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:06.446435 24166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:06.446854 24160 server_base.cc:1034] running on GCE node
I20260926 02:53:06.447494 24160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:06.448451 24160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:06.449692 24160 hybrid_clock.cc:648] HybridClock initialized: now 1790391186449632 us; error 61 us; skew 500 ppm
I20260926 02:53:06.450024 24160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:06 dist-test-slave-ktng krb5kdc[23784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391186, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:06.457607 24160 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:53:06.459128 24160 webserver.cc:460] Webserver started at http://127.19.185.195:34849/ using document root <none> and password file <none>
I20260926 02:53:06.459893 24160 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:06.460039 24160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:06.460350 24160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:06.463218 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "1eea6f1b001b4adaa2173d7e416dd719"
format_stamp: "Formatted at 2026-09-26 02:53:06 on dist-test-slave-ktng"
server_key: "b0e0195e0dd1d750d79abd157b955e90"
server_key_iv: "9425a8eb0f899b2dddf5af0896ebada6"
server_key_version: "encryptionkey@0"
I20260926 02:53:06.464044 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "1eea6f1b001b4adaa2173d7e416dd719"
format_stamp: "Formatted at 2026-09-26 02:53:06 on dist-test-slave-ktng"
server_key: "b0e0195e0dd1d750d79abd157b955e90"
server_key_iv: "9425a8eb0f899b2dddf5af0896ebada6"
server_key_version: "encryptionkey@0"
I20260926 02:53:06.469102 24160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:53:06.472409 24175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:06.473589 24160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:53:06.473755 24160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "1eea6f1b001b4adaa2173d7e416dd719"
format_stamp: "Formatted at 2026-09-26 02:53:06 on dist-test-slave-ktng"
server_key: "b0e0195e0dd1d750d79abd157b955e90"
server_key_iv: "9425a8eb0f899b2dddf5af0896ebada6"
server_key_version: "encryptionkey@0"
I20260926 02:53:06.473953 24160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:06.492476 24160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:06.496609 24160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:06.496934 24160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:06.497871 24160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:06.499167 24160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:06.499267 24160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:06.499452 24160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:06.499524 24160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:06.532670 24160 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:36283
I20260926 02:53:06.532706 24289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:36283 every 8 connection(s)
I20260926 02:53:06.533943 24160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:53:06.541728 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 24160
I20260926 02:53:06.541936 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:53:06.542368 20199 external_mini_cluster.cc:1428] Setting key 9aca337427fbfd7afdb0973f51bf74ba
Sep 26 02:53:06 dist-test-slave-ktng krb5kdc[23784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391186, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:06.557776 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:06.536536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:49823 (local address 127.19.185.254:39473)
0926 02:53:06.536729 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:06.536734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:06.537881 (+ 1147us) server_negotiation.cc:407] Connection header received
0926 02:53:06.539749 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:06.539754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:06.539847 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:06.539956 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:06.541416 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.542271 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.544244 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.544672 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.544806 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:06.552206 (+ 7400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:06.552211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:06.552214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:06.552246 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:06.554589 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:06.555260 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:06.555264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:06.555266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:06.555430 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:06.555921 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:06.555925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:06.555927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:06.556234 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:06.556393 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:06.557209 (+ 816us) server_negotiation.cc:299] Negotiation successful
0926 02:53:06.557451 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:53:06.558707 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:06.537043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39473 (local address 127.19.185.195:49823)
0926 02:53:06.537732 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:06.537769 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:06.539482 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:06.540190 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:06.540198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:06.540701 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:06.541235 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:06.541251 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.542405 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.542417 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:06.544097 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:06.544111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.544887 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.544896 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:06.545146 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:06.549817 (+ 4671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:06.549843 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:06.551994 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:06.554709 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:06.554716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:06.554726 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:06.555086 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:06.555556 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:06.555562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:06.555568 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:06.555783 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:06.556380 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:06.556393 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:06.556787 (+ 394us) client_negotiation.cc:769] Sending connection context
0926 02:53:06.557099 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 02:53:06.557555 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":270,"threads_started":1}
I20260926 02:53:06.559980 24290 heartbeater.cc:347] Connected to a master server at 127.19.185.254:39473
I20260926 02:53:06.560268 24290 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:06.560942 24290 heartbeater.cc:511] Master 127.19.185.254:39473 requested a full tablet report, sending...
I20260926 02:53:06.562243 23829 ts_manager.cc:194] Registered new tserver with Master: 1eea6f1b001b4adaa2173d7e416dd719 (127.19.185.195:36283)
I20260926 02:53:06.563031 23829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:49823
I20260926 02:53:06.569764 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:53:06.579334 24020 heartbeater.cc:503] Master 127.19.185.254:39473 was elected leader, sending a full tablet report...
Sep 26 02:53:06 dist-test-slave-ktng krb5kdc[23784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:53:06 dist-test-slave-ktng krb5kdc[23784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:53:06.619079 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:06.604367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34150 (local address 127.19.185.193:35383)
0926 02:53:06.604958 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:06.604965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:06.604993 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:53:06.605053 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:06.605059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:06.605287 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:06.605417 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:06.606832 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.607913 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.609119 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.609693 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.609790 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:06.613650 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:06.613661 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:06.613676 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:06.613709 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:06.616484 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:06.617112 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:06.617119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:06.617121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:06.617241 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:06.617663 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:06.617666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:06.617668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:06.617999 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:06.618250 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:06.618501 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 02:53:06.618683 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":291,"threads_started":1}
Sep 26 02:53:06 dist-test-slave-ktng krb5kdc[23784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:06.635893 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:06.623484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53002 (local address 127.19.185.254:39473)
0926 02:53:06.623716 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:06.623722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:06.623901 (+ 179us) server_negotiation.cc:407] Connection header received
0926 02:53:06.624010 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:06.624014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:06.624074 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:06.624158 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:06.625285 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.625988 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.627162 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.627658 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.627756 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:06.631360 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:06.631366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:06.631369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:06.631399 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:06.633681 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:06.634221 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:06.634224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:06.634226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:06.634336 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:06.634631 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:06.634634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:06.634636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:06.634912 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:06.635077 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:06.635404 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 02:53:06.635546 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260926 02:53:06.636480 23829 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53002
I20260926 02:53:06.637506 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 23890
I20260926 02:53:06.648963 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:06.638328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.19.185.193:35383)
0926 02:53:06.638480 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:06.638485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:06.638758 (+ 273us) server_negotiation.cc:407] Connection header received
0926 02:53:06.638875 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:06.638879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:06.638965 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:06.639042 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:06.640237 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.640936 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.642062 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.642445 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.642537 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:06.644628 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:06.644633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:06.644636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:06.644664 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:06.646944 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:06.647558 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:06.647561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:06.647564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:06.647674 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:06.647974 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:06.647977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:06.647979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:06.648225 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:06.648390 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:06.648594 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 02:53:06.648742 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:53:06.649575 24015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:06.731973 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 23890
I20260926 02:53:06.748051 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 24026
Sep 26 02:53:06 dist-test-slave-ktng krb5kdc[23784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:53:06.763936 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:06.749174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.19.185.194:38263)
0926 02:53:06.749710 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:06.749716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:06.749741 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:53:06.749801 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:06.749808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:06.750015 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:06.750151 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:06.751375 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.752351 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.753549 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.754051 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.754149 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:06.758394 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:06.758405 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:06.758419 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:06.758449 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:06.761196 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:06.761828 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:06.761834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:06.761837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:06.761958 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:06.762341 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:06.762345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:06.762347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:06.762696 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:06.762925 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:06.763243 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 02:53:06.763652 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":295,"threads_started":1}
I20260926 02:53:06.765225 24150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:06.848412 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 24026
I20260926 02:53:06.864806 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 24160
Sep 26 02:53:06 dist-test-slave-ktng krb5kdc[23784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:53:06.881901 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:06.865841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51296 (local address 127.19.185.195:36283)
0926 02:53:06.866481 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:06.866488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:06.866539 (+ 51us) server_negotiation.cc:407] Connection header received
0926 02:53:06.866606 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:06.866613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:06.866826 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:06.867006 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:06.868378 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.869506 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.870786 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:06.871411 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:06.871547 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:06.876049 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:06.876061 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:06.876095 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:06.876137 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:06.879178 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:06.879873 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:06.879880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:06.879882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:06.879997 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:06.880350 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:06.880361 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:06.880364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:06.880687 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:06.880992 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:06.881295 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 02:53:06.881556 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":356,"threads_started":1}
I20260926 02:53:06.883239 24285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:06.964814 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 24160
I20260926 02:53:06.980845 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 23797
I20260926 02:53:06.982226 23859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:07.084854 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 23797
2026-09-26T02:53:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3871 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:53:07 dist-test-slave-ktng krb5kdc[24309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:53:07 dist-test-slave-ktng krb5kdc[24309](info): set up 2 sockets
Sep 26 02:53:07 dist-test-slave-ktng krb5kdc[24309](info): commencing operation
krb5kdc: starting...
W20260926 02:53:08.178716 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:53:08 dist-test-slave-ktng krb5kdc[24309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:53:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:53:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:53:08.387127 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:35567
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:35567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:08.709074 24322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:08.709354 24322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:08.709542 24322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:08.716671 24322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:08.716782 24322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:08.716894 24322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:08.717001 24322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:08.717092 24322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:53:08.727015 24322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:35567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:35567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:08.728344 24322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:08.729651 24322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:08.741667 24328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:08.741727 24330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:08.741755 24327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:08.742645 24322 server_base.cc:1034] running on GCE node
I20260926 02:53:08.743470 24322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:08.744794 24322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:08.746026 24322 hybrid_clock.cc:648] HybridClock initialized: now 1790391188745981 us; error 42 us; skew 500 ppm
I20260926 02:53:08.746364 24322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:08 dist-test-slave-ktng krb5kdc[24309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391188, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:08.749454 24322 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:53:08.750663 24322 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:08.750794 24322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:08.751127 24322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:08.754048 24322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "db9bcf298215461a980e947625ccf563"
format_stamp: "Formatted at 2026-09-26 02:53:08 on dist-test-slave-ktng"
server_key: "7f2794a990f479393935265763eb29c7"
server_key_iv: "2fcf5bc834dce92b7ab29dff8c5c72a9"
server_key_version: "encryptionkey@0"
I20260926 02:53:08.754818 24322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "db9bcf298215461a980e947625ccf563"
format_stamp: "Formatted at 2026-09-26 02:53:08 on dist-test-slave-ktng"
server_key: "7f2794a990f479393935265763eb29c7"
server_key_iv: "2fcf5bc834dce92b7ab29dff8c5c72a9"
server_key_version: "encryptionkey@0"
I20260926 02:53:08.759954 24322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 02:53:08.763399 24336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:08.765108 24322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:53:08.765292 24322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "db9bcf298215461a980e947625ccf563"
format_stamp: "Formatted at 2026-09-26 02:53:08 on dist-test-slave-ktng"
server_key: "7f2794a990f479393935265763eb29c7"
server_key_iv: "2fcf5bc834dce92b7ab29dff8c5c72a9"
server_key_version: "encryptionkey@0"
I20260926 02:53:08.765511 24322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:08.789853 24322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:08.793891 24322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:08.794184 24322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:08.812664 24322 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:35567
I20260926 02:53:08.812685 24387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:35567 every 8 connection(s)
I20260926 02:53:08.813668 24322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:53:08.815907 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 24322
I20260926 02:53:08.816136 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:53:08.816962 20199 external_mini_cluster.cc:1428] Setting key 550dbe83bade5313131f0c7d49c103ed
I20260926 02:53:08.818327 24388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:53:08 dist-test-slave-ktng krb5kdc[24309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:08.832628 24388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563: Bootstrap starting.
I20260926 02:53:08.836591 24388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:08.837611 24388 log.cc:824] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:08.840338 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:08.819021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.19.185.254:35567)
0926 02:53:08.819835 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:08.819845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:08.819885 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:53:08.821385 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:08.821406 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:08.821786 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:08.822124 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:08.824009 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:08.825365 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:08.827089 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:08.827826 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:08.827957 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:08.832622 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:08.832635 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:08.832689 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:08.832727 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:08.836291 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:08.837036 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:08.837043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:08.837049 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:08.837205 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:08.837573 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:08.837579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:08.837582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:08.838175 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:08.838483 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:08.838953 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 02:53:08.839402 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":392,"threads_started":1}
I20260926 02:53:08.840593 24388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563: No bootstrap required, opened a new log
I20260926 02:53:08.848348 24388 raft_consensus.cc:348] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9bcf298215461a980e947625ccf563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 35567 } }
I20260926 02:53:08.848778 24388 raft_consensus.cc:374] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:08.848886 24388 raft_consensus.cc:729] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9bcf298215461a980e947625ccf563, State: Initialized, Role: FOLLOWER
I20260926 02:53:08.849475 24388 consensus_queue.cc:260] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9bcf298215461a980e947625ccf563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 35567 } }
I20260926 02:53:08.849758 24388 raft_consensus.cc:388] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:53:08.849881 24388 raft_consensus.cc:482] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:53:08.850008 24388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:08.852234 24388 raft_consensus.cc:504] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9bcf298215461a980e947625ccf563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 35567 } }
I20260926 02:53:08.852664 24388 leader_election.cc:302] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db9bcf298215461a980e947625ccf563; no voters:
I20260926 02:53:08.853147 24388 leader_election.cc:288] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:53:08.853298 24393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:08.854012 24393 raft_consensus.cc:686] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [term 1 LEADER]: Becoming Leader. State: Replica: db9bcf298215461a980e947625ccf563, State: Running, Role: LEADER
I20260926 02:53:08.854511 24393 consensus_queue.cc:237] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9bcf298215461a980e947625ccf563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 35567 } }
I20260926 02:53:08.855384 24388 sys_catalog.cc:564] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:08.858007 24394 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db9bcf298215461a980e947625ccf563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9bcf298215461a980e947625ccf563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 35567 } } }
I20260926 02:53:08.858474 24394 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:08.859858 24401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:53:08.859750 24395 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db9bcf298215461a980e947625ccf563. Latest consensus state: current_term: 1 leader_uuid: "db9bcf298215461a980e947625ccf563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9bcf298215461a980e947625ccf563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 35567 } } }
I20260926 02:53:08.860280 24395 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:08.865036 24401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:53:08.874496 24401 catalog_manager.cc:1347] Generated new cluster ID: db55ddd7983645e0b2356beea69117f2
I20260926 02:53:08.874600 24401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:53:08.909228 24401 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:53:08.911181 24401 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:53:08.931550 24401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563: Generated new TSK 0
I20260926 02:53:08.932473 24401 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:53:08.991487 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:35567
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:09.323477 24414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:09.323762 24414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:09.323858 24414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:09.331061 24414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:09.331182 24414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:09.331367 24414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:53:09.342085 24414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:35567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:09.343446 24414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:09.344731 24414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:09.358057 24420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:09.357980 24421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:09.358028 24423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:09.358978 24414 server_base.cc:1034] running on GCE node
I20260926 02:53:09.359688 24414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:09.360615 24414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:09.361831 24414 hybrid_clock.cc:648] HybridClock initialized: now 1790391189361774 us; error 48 us; skew 500 ppm
I20260926 02:53:09.362149 24414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:09 dist-test-slave-ktng krb5kdc[24309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391189, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:09.365381 24414 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:53:09.366665 24414 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:09.366797 24414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:09.367120 24414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:09.370217 24414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "b8f117ec15af467d9589d3e46c01b290"
format_stamp: "Formatted at 2026-09-26 02:53:09 on dist-test-slave-ktng"
server_key: "141650e7a557d1c085fe3b3870ad2fed"
server_key_iv: "8e847d1632d6830c5ba654a2d4c5a12a"
server_key_version: "encryptionkey@0"
I20260926 02:53:09.371006 24414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "b8f117ec15af467d9589d3e46c01b290"
format_stamp: "Formatted at 2026-09-26 02:53:09 on dist-test-slave-ktng"
server_key: "141650e7a557d1c085fe3b3870ad2fed"
server_key_iv: "8e847d1632d6830c5ba654a2d4c5a12a"
server_key_version: "encryptionkey@0"
I20260926 02:53:09.376564 24414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 02:53:09.379981 24429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:09.381338 24414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:53:09.381518 24414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "b8f117ec15af467d9589d3e46c01b290"
format_stamp: "Formatted at 2026-09-26 02:53:09 on dist-test-slave-ktng"
server_key: "141650e7a557d1c085fe3b3870ad2fed"
server_key_iv: "8e847d1632d6830c5ba654a2d4c5a12a"
server_key_version: "encryptionkey@0"
I20260926 02:53:09.381740 24414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:09.413687 24414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:09.417948 24414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:09.418236 24414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:09.419209 24414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:09.420552 24414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:09.420671 24414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:09.420799 24414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:09.420888 24414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:09.454367 24414 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:44961
I20260926 02:53:09.454453 24541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:44961 every 8 connection(s)
I20260926 02:53:09.455364 24414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:53:09.464092 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 24414
I20260926 02:53:09.464319 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:53:09.465102 20199 external_mini_cluster.cc:1428] Setting key 3e3c7acd8f7dfbeaafd411125a8705c7
Sep 26 02:53:09 dist-test-slave-ktng krb5kdc[24309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391189, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:09.476737 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:09.457807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:46101 (local address 127.19.185.254:35567)
0926 02:53:09.458357 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:09.458362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:09.459184 (+ 822us) server_negotiation.cc:407] Connection header received
0926 02:53:09.460940 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:09.460946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:09.461042 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:09.461172 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:09.462684 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:09.463469 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:09.465187 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:09.465576 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:09.465748 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:09.469858 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:09.469867 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:09.469874 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:09.469940 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:09.473365 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:09.474356 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:09.474360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:09.474362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:09.474484 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:09.474959 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:09.474962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:09.474965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:09.475232 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:09.475429 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:09.476304 (+ 875us) server_negotiation.cc:299] Negotiation successful
0926 02:53:09.476477 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":353,"threads_started":1}
I20260926 02:53:09.478266 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:09.458277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:35567 (local address 127.19.185.193:46101)
0926 02:53:09.458960 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:09.459001 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:09.460681 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:09.461402 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:09.461409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:09.461906 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:09.462443 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:09.462460 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:09.463626 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:09.463637 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:09.465040 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:09.465052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:09.465850 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:09.465859 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:09.466191 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:09.466889 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:09.466927 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:09.469634 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:09.473516 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:09.473528 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:09.473547 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:09.474206 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:09.474606 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:09.474613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:09.474619 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:09.474843 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:09.475417 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:09.475428 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:09.475880 (+ 452us) client_negotiation.cc:769] Sending connection context
0926 02:53:09.476214 (+ 334us) client_negotiation.cc:240] Negotiation successful
0926 02:53:09.476781 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":281,"threads_started":1}
I20260926 02:53:09.479984 24542 heartbeater.cc:347] Connected to a master server at 127.19.185.254:35567
I20260926 02:53:09.480329 24542 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:09.481139 24542 heartbeater.cc:511] Master 127.19.185.254:35567 requested a full tablet report, sending...
I20260926 02:53:09.483713 24353 ts_manager.cc:194] Registered new tserver with Master: b8f117ec15af467d9589d3e46c01b290 (127.19.185.193:44961)
I20260926 02:53:09.485877 24353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:46101
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:53:09.515076 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:35567
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:09.837271 24549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:09.837553 24549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:09.837649 24549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:09.844794 24549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:09.844900 24549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:09.845016 24549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:53:09.854573 24549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:35567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:09.855906 24549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:09.857224 24549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:09.869910 24554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:09.869813 24557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:09.869891 24555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:09.870750 24549 server_base.cc:1034] running on GCE node
I20260926 02:53:09.871398 24549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:09.872295 24549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:09.873495 24549 hybrid_clock.cc:648] HybridClock initialized: now 1790391189873448 us; error 40 us; skew 500 ppm
I20260926 02:53:09.873816 24549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:09 dist-test-slave-ktng krb5kdc[24309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391189, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:09.876909 24549 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:53:09.878077 24549 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:09.878162 24549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:09.878449 24549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:09.881280 24549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "b585413e886c42398b61f5222dbc5e70"
format_stamp: "Formatted at 2026-09-26 02:53:09 on dist-test-slave-ktng"
server_key: "0f27a614246902454ee5759d8024a79d"
server_key_iv: "9eede6601e810f32fe4da5bc8cd656c2"
server_key_version: "encryptionkey@0"
I20260926 02:53:09.882064 24549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "b585413e886c42398b61f5222dbc5e70"
format_stamp: "Formatted at 2026-09-26 02:53:09 on dist-test-slave-ktng"
server_key: "0f27a614246902454ee5759d8024a79d"
server_key_iv: "9eede6601e810f32fe4da5bc8cd656c2"
server_key_version: "encryptionkey@0"
I20260926 02:53:09.887218 24549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:53:09.890586 24563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:09.891992 24549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:53:09.892159 24549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "b585413e886c42398b61f5222dbc5e70"
format_stamp: "Formatted at 2026-09-26 02:53:09 on dist-test-slave-ktng"
server_key: "0f27a614246902454ee5759d8024a79d"
server_key_iv: "9eede6601e810f32fe4da5bc8cd656c2"
server_key_version: "encryptionkey@0"
I20260926 02:53:09.892371 24549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:09.918093 24549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:09.922871 24549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:09.923182 24549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:09.924335 24549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:09.925753 24549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:09.925880 24549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:09.926034 24549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:09.926133 24549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:09.960364 24549 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:34021
I20260926 02:53:09.960420 24675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:34021 every 8 connection(s)
I20260926 02:53:09.961259 24549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:53:09.964574 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 24549
I20260926 02:53:09.964797 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:53:09.965466 20199 external_mini_cluster.cc:1428] Setting key 250d8c3e0e43286f64cf5fb7aa0e8db7
Sep 26 02:53:09 dist-test-slave-ktng krb5kdc[24309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391189, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:09.983213 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:09.963706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:45549 (local address 127.19.185.254:35567)
0926 02:53:09.963880 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:09.963884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:09.965260 (+ 1376us) server_negotiation.cc:407] Connection header received
0926 02:53:09.967531 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:09.967536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:09.967619 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:09.967698 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:09.969837 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:09.970654 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:09.973143 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:09.973572 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:09.973735 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:09.977341 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:09.977346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:09.977350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:09.977380 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:09.980156 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:09.980898 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:09.980903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:09.980906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:09.981085 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:09.981521 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:09.981526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:09.981529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:09.981799 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:09.981958 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:09.982637 (+ 679us) server_negotiation.cc:299] Negotiation successful
0926 02:53:09.982860 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:53:09.984313 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:09.964255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:35567 (local address 127.19.185.194:45549)
0926 02:53:09.965052 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:09.965134 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:09.967213 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:09.967967 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:09.967980 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:09.968789 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:09.969654 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:09.969684 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:09.970797 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:09.970805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:09.972963 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:09.972980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:09.973835 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:09.973842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:09.974061 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:09.974744 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:09.974766 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:09.977115 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:09.980310 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:09.980319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:09.980338 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:09.980761 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:09.981211 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:09.981217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:09.981221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:09.981403 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:09.981926 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:09.981935 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:09.982280 (+ 345us) client_negotiation.cc:769] Sending connection context
0926 02:53:09.982540 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 02:53:09.983085 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":329,"threads_started":1}
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
I20260926 02:53:09.986044 24676 heartbeater.cc:347] Connected to a master server at 127.19.185.254:35567
I20260926 02:53:09.986403 24676 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:09.987213 24676 heartbeater.cc:511] Master 127.19.185.254:35567 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
I20260926 02:53:09.988929 24353 ts_manager.cc:194] Registered new tserver with Master: b585413e886c42398b61f5222dbc5e70 (127.19.185.194:34021)
I20260926 02:53:09.989769 24353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:45549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:53:10.012785 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:35567
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:10.334254 24681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:10.334537 24681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:10.334641 24681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:10.342391 24681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:10.342499 24681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:10.342614 24681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:53:10.352311 24681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:35567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:10.353528 24681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:10.354768 24681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:10.367424 24686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:10.367496 24689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:10.367465 24687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:10.368088 24681 server_base.cc:1034] running on GCE node
I20260926 02:53:10.368695 24681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:10.370464 24681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:10.371665 24681 hybrid_clock.cc:648] HybridClock initialized: now 1790391190371609 us; error 69 us; skew 500 ppm
I20260926 02:53:10.371975 24681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:10 dist-test-slave-ktng krb5kdc[24309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391190, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:10.375061 24681 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:53:10.376291 24681 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:10.376407 24681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:10.376749 24681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:10.379678 24681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "5320abcb9f92458f81f6c8e346f7116a"
format_stamp: "Formatted at 2026-09-26 02:53:10 on dist-test-slave-ktng"
server_key: "abc30c19022a8e8b7fdaad6329b7a947"
server_key_iv: "0b773dd09149c6611519860e38ee2de5"
server_key_version: "encryptionkey@0"
I20260926 02:53:10.380565 24681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "5320abcb9f92458f81f6c8e346f7116a"
format_stamp: "Formatted at 2026-09-26 02:53:10 on dist-test-slave-ktng"
server_key: "abc30c19022a8e8b7fdaad6329b7a947"
server_key_iv: "0b773dd09149c6611519860e38ee2de5"
server_key_version: "encryptionkey@0"
I20260926 02:53:10.385599 24681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:53:10.389159 24695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:10.390602 24681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:53:10.390780 24681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "5320abcb9f92458f81f6c8e346f7116a"
format_stamp: "Formatted at 2026-09-26 02:53:10 on dist-test-slave-ktng"
server_key: "abc30c19022a8e8b7fdaad6329b7a947"
server_key_iv: "0b773dd09149c6611519860e38ee2de5"
server_key_version: "encryptionkey@0"
I20260926 02:53:10.390991 24681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:10.412379 24681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:10.416821 24681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:10.417125 24681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:10.418134 24681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:10.419575 24681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:10.419709 24681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:10.419866 24681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:10.419955 24681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:10.454113 24681 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:42653
I20260926 02:53:10.454154 24808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:42653 every 8 connection(s)
I20260926 02:53:10.455159 24681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:53:10.461786 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 24681
I20260926 02:53:10.461993 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:53:10.462424 20199 external_mini_cluster.cc:1428] Setting key 81e926332800a4a155f08749039d836d
Sep 26 02:53:10 dist-test-slave-ktng krb5kdc[24309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391190, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:10.475751 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.457913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:57555 (local address 127.19.185.254:35567)
0926 02:53:10.458101 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:10.458107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:10.459490 (+ 1383us) server_negotiation.cc:407] Connection header received
0926 02:53:10.461288 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:10.461295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:10.461441 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:10.461559 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:10.463513 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.464537 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.466474 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.466868 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.466962 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.470383 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:10.470388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:10.470391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:10.470420 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:10.473064 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:10.473717 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:10.473721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:10.473723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:10.473842 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:10.474246 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:10.474249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:10.474251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:10.474495 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:10.474606 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:10.475264 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 02:53:10.475506 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:53:10.476886 24811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.458554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:35567 (local address 127.19.185.195:57555)
0926 02:53:10.459274 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:10.459380 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:10.460987 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:10.461768 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:10.461776 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:10.462419 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:10.463270 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:10.463335 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.464699 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.464704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:10.466342 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:10.466354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.467045 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.467049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:10.467274 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.468040 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:10.468062 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:10.470213 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:10.473191 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:10.473197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:10.473209 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:10.473609 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:10.473931 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:10.473934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:10.473936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:10.474095 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:10.474594 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:10.474599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:10.474890 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 02:53:10.475145 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 02:53:10.475584 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":294,"threads_started":1}
I20260926 02:53:10.478302 24809 heartbeater.cc:347] Connected to a master server at 127.19.185.254:35567
I20260926 02:53:10.478554 24809 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:10.479121 24809 heartbeater.cc:511] Master 127.19.185.254:35567 requested a full tablet report, sending...
I20260926 02:53:10.480605 24352 ts_manager.cc:194] Registered new tserver with Master: 5320abcb9f92458f81f6c8e346f7116a (127.19.185.195:42653)
I20260926 02:53:10.481431 24352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:57555
I20260926 02:53:10.489135 24542 heartbeater.cc:503] Master 127.19.185.254:35567 was elected leader, sending a full tablet report...
I20260926 02:53:10.490573 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:53:10.508378 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.495360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.19.185.254:35567)
0926 02:53:10.495581 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:10.495589 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:10.495972 (+ 383us) server_negotiation.cc:407] Connection header received
0926 02:53:10.496195 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:10.496200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:10.496286 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:10.496390 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:10.497872 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.498626 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.499960 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.500382 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.500522 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.502706 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:10.502711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:10.502714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:10.502745 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:10.505375 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:10.506200 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:10.506205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:10.506207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:10.506355 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:10.506779 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:10.506783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:10.506785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:10.507162 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:10.507555 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:10.507729 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 02:53:10.507910 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:53:10.512816 24352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:53:10.519687 24352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:53:10.557753 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.533249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.19.185.195:42653)
0926 02:53:10.546979 (+ 13730us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:10.546989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:10.547027 (+ 38us) server_negotiation.cc:407] Connection header received
0926 02:53:10.547142 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:10.547150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:10.547490 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:10.548483 (+ 993us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:10.548871 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.550883 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.554544 (+ 3661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.555985 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.556180 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.556372 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:10.557223 (+ 851us) server_negotiation.cc:299] Negotiation successful
0926 02:53:10.557392 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13551,"thread_start_us":13386,"threads_started":1}
I20260926 02:53:10.557822 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.532947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:42653 (local address 127.0.0.1:36648)
0926 02:53:10.546032 (+ 13085us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:10.546071 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:10.546298 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:10.547753 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:10.547764 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:10.547797 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:10.548249 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:10.548269 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.551091 (+ 2822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.551100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:10.554356 (+ 3256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:10.554370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.556532 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.556538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:10.556825 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.556916 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 02:53:10.557042 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 02:53:10.557272 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12822,"mutex_wait_us":145,"thread_start_us":14105,"threads_started":1}
I20260926 02:53:10.558279 24824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.533781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40870 (local address 127.19.185.194:34021)
0926 02:53:10.545538 (+ 11757us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:10.545552 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:10.547628 (+ 2076us) server_negotiation.cc:407] Connection header received
0926 02:53:10.547743 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:10.547754 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:10.548039 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:10.548257 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:10.549316 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.551260 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.554642 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.556116 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.556269 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.556429 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:10.557797 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0926 02:53:10.557975 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11505,"thread_start_us":11384,"threads_started":1}
I20260926 02:53:10.558576 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.533191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:34021 (local address 127.0.0.1:40870)
0926 02:53:10.547352 (+ 14161us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:10.547373 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:10.547501 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:10.548548 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:10.548556 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:10.548590 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:10.549092 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:10.549107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.551495 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.551503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:10.554482 (+ 2979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:10.554504 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.556590 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.556599 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:10.556831 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.556914 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 02:53:10.557040 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 02:53:10.557241 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14004,"mutex_wait_us":54,"thread_start_us":12433,"threads_started":1}
I20260926 02:53:10.562207 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.547527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:44961 (local address 127.0.0.1:54016)
0926 02:53:10.554209 (+ 6682us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:10.554226 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:10.554362 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:10.554995 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:10.554999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:10.555024 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:10.555458 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:10.555470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.557399 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.557406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:10.559802 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:10.559813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.561488 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.561496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:10.561652 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.561681 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 02:53:10.561767 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 02:53:10.561892 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6563,"thread_start_us":6421,"threads_started":1}
I20260926 02:53:10.562278 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.547885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.19.185.193:44961)
0926 02:53:10.552642 (+ 4757us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:10.552651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:10.554349 (+ 1698us) server_negotiation.cc:407] Connection header received
0926 02:53:10.554528 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:10.554539 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:10.554785 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:10.554992 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:10.555615 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.557241 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.559948 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.561295 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.561517 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.561676 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:10.561832 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 02:53:10.561979 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4592,"thread_start_us":4445,"threads_started":1}
I20260926 02:53:10.562197 24611 tablet_service.cc:1467] Processing CreateTablet for tablet 3bdde0d3cfce45438be87812529e6ac8 (DEFAULT_TABLE table=test-table [id=bad08565c57a40a28e6d10f9c0baf52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:10.562197 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 3bdde0d3cfce45438be87812529e6ac8 (DEFAULT_TABLE table=test-table [id=bad08565c57a40a28e6d10f9c0baf52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:10.563720 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bdde0d3cfce45438be87812529e6ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:10.563740 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bdde0d3cfce45438be87812529e6ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:10.566155 24477 tablet_service.cc:1467] Processing CreateTablet for tablet 3bdde0d3cfce45438be87812529e6ac8 (DEFAULT_TABLE table=test-table [id=bad08565c57a40a28e6d10f9c0baf52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:10.567353 24477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bdde0d3cfce45438be87812529e6ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:10.578970 24827 tablet_bootstrap.cc:492] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290: Bootstrap starting.
I20260926 02:53:10.582012 24827 tablet_bootstrap.cc:654] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:10.582989 24827 log.cc:824] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:10.583963 24829 tablet_bootstrap.cc:492] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a: Bootstrap starting.
I20260926 02:53:10.584928 24828 tablet_bootstrap.cc:492] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70: Bootstrap starting.
I20260926 02:53:10.587260 24827 tablet_bootstrap.cc:492] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290: No bootstrap required, opened a new log
I20260926 02:53:10.587649 24827 ts_tablet_manager.cc:1397] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 02:53:10.588613 24828 tablet_bootstrap.cc:654] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:10.588613 24829 tablet_bootstrap.cc:654] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:10.589877 24829 log.cc:824] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:10.589877 24828 log.cc:824] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:10.592890 24828 tablet_bootstrap.cc:492] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70: No bootstrap required, opened a new log
I20260926 02:53:10.593199 24828 ts_tablet_manager.cc:1397] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:53:10.593206 24829 tablet_bootstrap.cc:492] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a: No bootstrap required, opened a new log
I20260926 02:53:10.593461 24829 ts_tablet_manager.cc:1397] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260926 02:53:10.597858 24827 raft_consensus.cc:348] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5320abcb9f92458f81f6c8e346f7116a" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42653 } } peers { permanent_uuid: "b585413e886c42398b61f5222dbc5e70" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34021 } } peers { permanent_uuid: "b8f117ec15af467d9589d3e46c01b290" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44961 } }
I20260926 02:53:10.598371 24827 raft_consensus.cc:374] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:10.598489 24827 raft_consensus.cc:729] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f117ec15af467d9589d3e46c01b290, State: Initialized, Role: FOLLOWER
I20260926 02:53:10.599174 24827 consensus_queue.cc:260] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5320abcb9f92458f81f6c8e346f7116a" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42653 } } peers { permanent_uuid: "b585413e886c42398b61f5222dbc5e70" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34021 } } peers { permanent_uuid: "b8f117ec15af467d9589d3e46c01b290" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44961 } }
I20260926 02:53:10.601008 24827 ts_tablet_manager.cc:1428] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 02:53:10.602088 24828 raft_consensus.cc:348] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5320abcb9f92458f81f6c8e346f7116a" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42653 } } peers { permanent_uuid: "b585413e886c42398b61f5222dbc5e70" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34021 } } peers { permanent_uuid: "b8f117ec15af467d9589d3e46c01b290" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44961 } }
I20260926 02:53:10.602437 24828 raft_consensus.cc:374] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:10.602563 24828 raft_consensus.cc:729] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b585413e886c42398b61f5222dbc5e70, State: Initialized, Role: FOLLOWER
I20260926 02:53:10.603132 24828 consensus_queue.cc:260] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5320abcb9f92458f81f6c8e346f7116a" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42653 } } peers { permanent_uuid: "b585413e886c42398b61f5222dbc5e70" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34021 } } peers { permanent_uuid: "b8f117ec15af467d9589d3e46c01b290" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44961 } }
I20260926 02:53:10.604233 24676 heartbeater.cc:503] Master 127.19.185.254:35567 was elected leader, sending a full tablet report...
I20260926 02:53:10.603840 24829 raft_consensus.cc:348] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5320abcb9f92458f81f6c8e346f7116a" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42653 } } peers { permanent_uuid: "b585413e886c42398b61f5222dbc5e70" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34021 } } peers { permanent_uuid: "b8f117ec15af467d9589d3e46c01b290" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44961 } }
I20260926 02:53:10.604372 24829 raft_consensus.cc:374] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:10.604465 24829 raft_consensus.cc:729] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5320abcb9f92458f81f6c8e346f7116a, State: Initialized, Role: FOLLOWER
I20260926 02:53:10.605168 24828 ts_tablet_manager.cc:1428] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70: Time spent starting tablet: real 0.012s user 0.006s sys 0.008s
I20260926 02:53:10.605031 24829 consensus_queue.cc:260] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5320abcb9f92458f81f6c8e346f7116a" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42653 } } peers { permanent_uuid: "b585413e886c42398b61f5222dbc5e70" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34021 } } peers { permanent_uuid: "b8f117ec15af467d9589d3e46c01b290" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44961 } }
I20260926 02:53:10.606099 24809 heartbeater.cc:503] Master 127.19.185.254:35567 was elected leader, sending a full tablet report...
I20260926 02:53:10.606812 24829 ts_tablet_manager.cc:1428] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260926 02:53:10.706911 24810 tablet.cc:2369] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:53:10.707715 24543 tablet.cc:2369] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:53:10.713075 24677 tablet.cc:2369] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:53:10.761811 24833 raft_consensus.cc:482] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:53:10.762137 24833 raft_consensus.cc:504] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5320abcb9f92458f81f6c8e346f7116a" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42653 } } peers { permanent_uuid: "b585413e886c42398b61f5222dbc5e70" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34021 } } peers { permanent_uuid: "b8f117ec15af467d9589d3e46c01b290" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44961 } }
I20260926 02:53:10.764164 24833 leader_election.cc:288] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5320abcb9f92458f81f6c8e346f7116a (127.19.185.195:42653), b585413e886c42398b61f5222dbc5e70 (127.19.185.194:34021)
I20260926 02:53:10.771332 24824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.764914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:38921 (local address 127.19.185.194:34021)
0926 02:53:10.765124 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:10.765130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:10.765682 (+ 552us) server_negotiation.cc:407] Connection header received
0926 02:53:10.765861 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:10.765864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:10.765920 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:10.766033 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:10.766677 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.767503 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.769448 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.770327 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.770508 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.770591 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:10.770911 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 02:53:10.771046 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:53:10.771355 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.764456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:52147 (local address 127.19.185.195:42653)
0926 02:53:10.764882 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:10.764892 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:10.765592 (+ 700us) server_negotiation.cc:407] Connection header received
0926 02:53:10.765805 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:10.765812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:10.765893 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:10.765958 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:10.766585 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.767398 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.769394 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.770325 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.770531 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.770615 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:10.770940 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 02:53:10.771066 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260926 02:53:10.771507 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.764450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:34021 (local address 127.19.185.193:38921)
0926 02:53:10.765530 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:10.765572 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:10.765734 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:10.766083 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:10.766087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:10.766115 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:10.766526 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:10.766541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.767638 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.767641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:10.769242 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:10.769276 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.770568 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.770574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:10.770732 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.770773 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 02:53:10.770873 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 02:53:10.771000 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":7,"thread_start_us":512,"threads_started":1}
I20260926 02:53:10.771524 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.764299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:42653 (local address 127.19.185.193:52147)
0926 02:53:10.765474 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:10.765490 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:10.765617 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:10.766011 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:10.766019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:10.766051 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:10.766435 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:10.766449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.767519 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.767524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:10.769247 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:10.769259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.770604 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.770610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:10.770783 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.770805 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 02:53:10.770912 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 02:53:10.771044 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":63,"thread_start_us":933,"threads_started":1}
I20260926 02:53:10.772864 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdde0d3cfce45438be87812529e6ac8" candidate_uuid: "b8f117ec15af467d9589d3e46c01b290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5320abcb9f92458f81f6c8e346f7116a" is_pre_election: true
I20260926 02:53:10.772886 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdde0d3cfce45438be87812529e6ac8" candidate_uuid: "b8f117ec15af467d9589d3e46c01b290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b585413e886c42398b61f5222dbc5e70" is_pre_election: true
I20260926 02:53:10.773438 24764 raft_consensus.cc:2454] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f117ec15af467d9589d3e46c01b290 in term 0.
I20260926 02:53:10.773458 24631 raft_consensus.cc:2454] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f117ec15af467d9589d3e46c01b290 in term 0.
I20260926 02:53:10.774230 24433 leader_election.cc:302] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5320abcb9f92458f81f6c8e346f7116a, b8f117ec15af467d9589d3e46c01b290; no voters:
I20260926 02:53:10.774688 24833 raft_consensus.cc:2781] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:53:10.774861 24833 raft_consensus.cc:482] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:53:10.774976 24833 raft_consensus.cc:3037] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:10.777206 24833 raft_consensus.cc:504] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5320abcb9f92458f81f6c8e346f7116a" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42653 } } peers { permanent_uuid: "b585413e886c42398b61f5222dbc5e70" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34021 } } peers { permanent_uuid: "b8f117ec15af467d9589d3e46c01b290" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44961 } }
I20260926 02:53:10.777765 24833 leader_election.cc:288] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [CANDIDATE]: Term 1 election: Requested vote from peers 5320abcb9f92458f81f6c8e346f7116a (127.19.185.195:42653), b585413e886c42398b61f5222dbc5e70 (127.19.185.194:34021)
I20260926 02:53:10.778229 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdde0d3cfce45438be87812529e6ac8" candidate_uuid: "b8f117ec15af467d9589d3e46c01b290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5320abcb9f92458f81f6c8e346f7116a"
I20260926 02:53:10.778267 24631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdde0d3cfce45438be87812529e6ac8" candidate_uuid: "b8f117ec15af467d9589d3e46c01b290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b585413e886c42398b61f5222dbc5e70"
I20260926 02:53:10.778460 24764 raft_consensus.cc:3037] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:10.778498 24631 raft_consensus.cc:3037] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:10.782065 24764 raft_consensus.cc:2454] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f117ec15af467d9589d3e46c01b290 in term 1.
I20260926 02:53:10.782083 24631 raft_consensus.cc:2454] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f117ec15af467d9589d3e46c01b290 in term 1.
I20260926 02:53:10.782516 24433 leader_election.cc:302] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5320abcb9f92458f81f6c8e346f7116a, b8f117ec15af467d9589d3e46c01b290; no voters:
I20260926 02:53:10.782855 24833 raft_consensus.cc:2781] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:10.783375 24833 raft_consensus.cc:686] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [term 1 LEADER]: Becoming Leader. State: Replica: b8f117ec15af467d9589d3e46c01b290, State: Running, Role: LEADER
I20260926 02:53:10.783954 24833 consensus_queue.cc:237] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5320abcb9f92458f81f6c8e346f7116a" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42653 } } peers { permanent_uuid: "b585413e886c42398b61f5222dbc5e70" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34021 } } peers { permanent_uuid: "b8f117ec15af467d9589d3e46c01b290" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44961 } }
I20260926 02:53:10.789204 24352 catalog_manager.cc:5495] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f117ec15af467d9589d3e46c01b290 (127.19.185.193). New cstate: current_term: 1 leader_uuid: "b8f117ec15af467d9589d3e46c01b290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5320abcb9f92458f81f6c8e346f7116a" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b585413e886c42398b61f5222dbc5e70" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f117ec15af467d9589d3e46c01b290" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44961 } health_report { overall_health: HEALTHY } } }
I20260926 02:53:10.855784 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.850002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.19.185.193:44961)
0926 02:53:10.850248 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:10.850254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:10.850373 (+ 119us) server_negotiation.cc:407] Connection header received
0926 02:53:10.850659 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:10.850665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:10.850754 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:10.850914 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:53:10.851416 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.852160 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.853539 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.853942 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.854071 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.854577 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:53:10.855096 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:53:10.855279 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:10.855527 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 02:53:10.855591 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 02:53:10.864089 24764 raft_consensus.cc:1261] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a [term 1 FOLLOWER]: Refusing update from remote peer b8f117ec15af467d9589d3e46c01b290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:10.864087 24631 raft_consensus.cc:1261] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70 [term 1 FOLLOWER]: Refusing update from remote peer b8f117ec15af467d9589d3e46c01b290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:10.865065 24838 consensus_queue.cc:1035] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5320abcb9f92458f81f6c8e346f7116a" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 42653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:10.865473 24833 consensus_queue.cc:1035] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b585413e886c42398b61f5222dbc5e70" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:10.879859 24841 mvcc.cc:205] Tried to move back new op lower bound from 7333442317769277440 to 7333442317454938112. Current Snapshot: MvccSnapshot[applied={T|T < 7333442317769277440}]
I20260926 02:53:10.894745 24353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:10.895040 24353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:10.898762 24353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db9bcf298215461a980e947625ccf563: Sending DeleteTablet for 3 replicas of tablet 3bdde0d3cfce45438be87812529e6ac8
I20260926 02:53:10.899966 24743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bdde0d3cfce45438be87812529e6ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:10 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:36648
I20260926 02:53:10.899977 24611 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bdde0d3cfce45438be87812529e6ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:10 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:40870
I20260926 02:53:10.900314 24477 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bdde0d3cfce45438be87812529e6ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:10 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:54016
I20260926 02:53:10.902477 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 24414
I20260926 02:53:10.913638 24850 tablet_replica.cc:330] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a: stopping tablet replica
I20260926 02:53:10.914173 24850 raft_consensus.cc:2229] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:10.914769 24850 raft_consensus.cc:2258] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:10.915712 24851 tablet_replica.cc:330] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70: stopping tablet replica
I20260926 02:53:10.916189 24851 raft_consensus.cc:2229] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:10.916751 24851 raft_consensus.cc:2258] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:10.919216 24851 ts_tablet_manager.cc:1905] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:10.921073 24850 ts_tablet_manager.cc:1905] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:10.925639 24852 tablet_replica.cc:330] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290: stopping tablet replica
I20260926 02:53:10.926134 24852 raft_consensus.cc:2229] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [term 1 LEADER]: Raft consensus shutting down.
Sep 26 02:53:10 dist-test-slave-ktng krb5kdc[24309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:53:10.927028 24852 raft_consensus.cc:2258] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:10.929481 24852 ts_tablet_manager.cc:1905] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:10.934653 24851 ts_tablet_manager.cc:1918] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:10.935992 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:10.904492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54044 (local address 127.19.185.193:44961)
0926 02:53:10.905133 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:10.905141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:10.909231 (+ 4090us) server_negotiation.cc:407] Connection header received
0926 02:53:10.909432 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:10.909440 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:10.909550 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:10.909702 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:10.917846 (+ 8144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.919528 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.920803 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:10.921823 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:10.921975 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:10.928004 (+ 6029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:10.928019 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:10.928039 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:10.928087 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:10.932279 (+ 4192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:10.933161 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:10.933187 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:10.933191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:10.933359 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:10.933863 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:10.933868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:10.933871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:10.934351 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:10.934694 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:10.935317 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 02:53:10.935597 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474}
I20260926 02:53:10.937388 24537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:10.938923 24850 ts_tablet_manager.cc:1918] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:10.939363 24850 log.cc:1196] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/3bdde0d3cfce45438be87812529e6ac8
I20260926 02:53:10.939443 24851 log.cc:1196] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/3bdde0d3cfce45438be87812529e6ac8
I20260926 02:53:10.940059 24851 ts_tablet_manager.cc:1939] T 3bdde0d3cfce45438be87812529e6ac8 P b585413e886c42398b61f5222dbc5e70: Deleting consensus metadata
I20260926 02:53:10.940122 24850 ts_tablet_manager.cc:1939] T 3bdde0d3cfce45438be87812529e6ac8 P 5320abcb9f92458f81f6c8e346f7116a: Deleting consensus metadata
I20260926 02:53:10.942189 24337 catalog_manager.cc:4841] TS b585413e886c42398b61f5222dbc5e70 (127.19.185.194:34021): tablet 3bdde0d3cfce45438be87812529e6ac8 (table test-table [id=bad08565c57a40a28e6d10f9c0baf52b]) successfully deleted
I20260926 02:53:10.942384 24340 catalog_manager.cc:4841] TS 5320abcb9f92458f81f6c8e346f7116a (127.19.185.195:42653): tablet 3bdde0d3cfce45438be87812529e6ac8 (table test-table [id=bad08565c57a40a28e6d10f9c0baf52b]) successfully deleted
I20260926 02:53:11.035075 24852 ts_tablet_manager.cc:1918] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:11.035378 24852 log.cc:1196] T 3bdde0d3cfce45438be87812529e6ac8 P b8f117ec15af467d9589d3e46c01b290: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/3bdde0d3cfce45438be87812529e6ac8
I20260926 02:53:11.035854 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 24414
W20260926 02:53:11.054134 24340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.185.193:44961 (error 108)
I20260926 02:53:11.054912 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 24549
I20260926 02:53:11.055960 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:11.055486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:44961 (local address 127.0.0.1:54054)
0926 02:53:11.055683 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:11.055806 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 02:53:11.056236 24340 catalog_manager.cc:4584] TS b8f117ec15af467d9589d3e46c01b290 (127.19.185.193:44961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3bdde0d3cfce45438be87812529e6ac8: Network error: Client connection negotiation failed: client connection to 127.19.185.193:44961: connect: Connection refused (error 111)
Sep 26 02:53:11 dist-test-slave-ktng krb5kdc[24309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:53:11.070578 24824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:11.055753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40884 (local address 127.19.185.194:34021)
0926 02:53:11.055920 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:11.055927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:11.055946 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:53:11.056175 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:11.056182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:11.056257 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:11.056341 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:11.057583 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:11.058496 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:11.059964 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:11.060366 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:11.060464 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:11.064495 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:11.064520 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:11.064536 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:11.064570 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:11.067614 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:11.068299 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:11.068306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:11.068309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:11.068441 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:11.068857 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:11.068861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:11.068864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:11.069403 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:11.069628 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:11.070049 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 02:53:11.070217 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:53:11.071404 24671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:11.110102 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:11.109558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:44961 (local address 127.0.0.1:54058)
0926 02:53:11.109793 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:11.109918 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 02:53:11.111212 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:11.110820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:44961 (local address 127.0.0.1:54068)
0926 02:53:11.110979 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:11.111061 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 02:53:11.159997 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 24549
I20260926 02:53:11.176414 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:11.175934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:44961 (local address 127.0.0.1:54072)
0926 02:53:11.176156 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:11.176243 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 02:53:11.177392 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:11.177076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:44961 (local address 127.0.0.1:54074)
0926 02:53:11.177217 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:11.177288 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 02:53:11.184731 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 24681
Sep 26 02:53:11 dist-test-slave-ktng krb5kdc[24309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:53:11.199604 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:11.185423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.19.185.195:42653)
0926 02:53:11.185628 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:11.185634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:11.185656 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:53:11.185767 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:11.185776 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:11.185847 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:11.185954 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:11.187237 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:11.188005 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:11.189234 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:11.189644 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:11.189817 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:11.194064 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:11.194081 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:11.194098 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:11.194131 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:11.196877 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:11.197530 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:11.197538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:11.197540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:11.197691 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:11.198089 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:11.198096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:11.198104 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:11.198440 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:11.198749 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:11.199022 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 02:53:11.199236 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:53:11.200214 24804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:11.257733 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:11.257206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:44961 (local address 127.0.0.1:54078)
0926 02:53:11.257436 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:11.257557 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 02:53:11.258900 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:11.258511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:44961 (local address 127.0.0.1:54090)
0926 02:53:11.258729 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:11.258795 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260926 02:53:11.288913 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 24681
I20260926 02:53:11.305263 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 24322
I20260926 02:53:11.306392 24383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:11.414176 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 24322
2026-09-26T02:53:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4328 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T02:53:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:53:11Z Disabled control of system clock
I20260926 02:53:11.481428 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:46677
--webserver_interface=127.19.185.254
--webserver_port=0
--builtin_ntp_servers=127.19.185.212:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:46677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 02:53:11.801111 24862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:11.801389 24862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:11.801545 24862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:11.808910 24862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:11.809015 24862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:11.809094 24862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:11.809192 24862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:11.809278 24862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:53:11.818804 24862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:46677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:46677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:11.820186 24862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:11.822041 24862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:11.833863 24867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:11.833884 24868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:11.834331 24870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:11.835287 24862 server_base.cc:1034] running on GCE node
I20260926 02:53:11.836107 24862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:11.837504 24862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:11.838721 24862 hybrid_clock.cc:648] HybridClock initialized: now 1790391191838665 us; error 44 us; skew 500 ppm
I20260926 02:53:11.839030 24862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:53:11.846699 24862 webserver.cc:460] Webserver started at http://127.19.185.254:44671/ using document root <none> and password file <none>
I20260926 02:53:11.847470 24862 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:11.847575 24862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:11.847854 24862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:11.850772 24862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "e7c342a388d84ce3834c34d94feb6d12"
format_stamp: "Formatted at 2026-09-26 02:53:11 on dist-test-slave-ktng"
server_key: "718676c6f637c58ee9015322988367ce"
server_key_iv: "dde49cfd560a280c39786b594ce883b2"
server_key_version: "encryptionkey@0"
I20260926 02:53:11.851636 24862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "e7c342a388d84ce3834c34d94feb6d12"
format_stamp: "Formatted at 2026-09-26 02:53:11 on dist-test-slave-ktng"
server_key: "718676c6f637c58ee9015322988367ce"
server_key_iv: "dde49cfd560a280c39786b594ce883b2"
server_key_version: "encryptionkey@0"
I20260926 02:53:11.856673 24862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:53:11.860167 24876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:11.861492 24862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:53:11.861689 24862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "e7c342a388d84ce3834c34d94feb6d12"
format_stamp: "Formatted at 2026-09-26 02:53:11 on dist-test-slave-ktng"
server_key: "718676c6f637c58ee9015322988367ce"
server_key_iv: "dde49cfd560a280c39786b594ce883b2"
server_key_version: "encryptionkey@0"
I20260926 02:53:11.861904 24862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:11.867627 24862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:11.868369 24862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:11.868666 24862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:11.886639 24862 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:46677
I20260926 02:53:11.886660 24928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:46677 every 8 connection(s)
I20260926 02:53:11.887986 24862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:53:11.889099 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 24862
I20260926 02:53:11.889340 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:53:11.890154 20199 external_mini_cluster.cc:1428] Setting key 5bac5cecdc1defa4c32b7908b2a94de4
I20260926 02:53:11.892910 24929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:11.898972 24932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:11.892282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.19.185.254:46677)
0926 02:53:11.893358 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:11.893369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:11.893414 (+ 45us) server_negotiation.cc:407] Connection header received
0926 02:53:11.895057 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:11.895080 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:11.895602 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:11.895950 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:11.897162 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:11.897169 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:11.897181 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:11.897201 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:11.897310 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:11.897328 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:11.897510 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:11.897646 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 02:53:11.897955 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":378,"threads_started":1}
I20260926 02:53:11.904439 24929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12: Bootstrap starting.
I20260926 02:53:11.907426 24929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:11.908428 24929 log.cc:824] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:11.911387 24929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12: No bootstrap required, opened a new log
I20260926 02:53:11.918491 24929 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c342a388d84ce3834c34d94feb6d12" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46677 } }
I20260926 02:53:11.918879 24929 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:11.918984 24929 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c342a388d84ce3834c34d94feb6d12, State: Initialized, Role: FOLLOWER
I20260926 02:53:11.919611 24929 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c342a388d84ce3834c34d94feb6d12" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46677 } }
I20260926 02:53:11.919853 24929 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:53:11.919991 24929 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:53:11.920176 24929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:11.922391 24929 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c342a388d84ce3834c34d94feb6d12" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46677 } }
I20260926 02:53:11.922914 24929 leader_election.cc:302] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7c342a388d84ce3834c34d94feb6d12; no voters:
I20260926 02:53:11.923604 24929 leader_election.cc:288] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:53:11.923772 24934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:11.924721 24934 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [term 1 LEADER]: Becoming Leader. State: Replica: e7c342a388d84ce3834c34d94feb6d12, State: Running, Role: LEADER
I20260926 02:53:11.925379 24934 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c342a388d84ce3834c34d94feb6d12" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46677 } }
I20260926 02:53:11.925773 24929 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:11.928309 24935 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7c342a388d84ce3834c34d94feb6d12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c342a388d84ce3834c34d94feb6d12" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46677 } } }
I20260926 02:53:11.928689 24935 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:11.929311 24936 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c342a388d84ce3834c34d94feb6d12. Latest consensus state: current_term: 1 leader_uuid: "e7c342a388d84ce3834c34d94feb6d12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c342a388d84ce3834c34d94feb6d12" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46677 } } }
I20260926 02:53:11.929646 24936 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:11.929718 24941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:53:11.936735 24941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:53:11.945379 24941 catalog_manager.cc:1347] Generated new cluster ID: 6faf7295c30d475cb618de3dc89381b4
I20260926 02:53:11.945488 24941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:53:11.956038 24941 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:53:11.956511 24941 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:53:11.966919 24941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12: Generated new TSK 0
I20260926 02:53:11.968079 24941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:53:11.984737 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46677
--builtin_ntp_servers=127.19.185.212:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 02:53:12.311929 24953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:12.312220 24953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:12.312383 24953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:12.319768 24953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:12.319877 24953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:12.320109 24953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:53:12.329349 24953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:12.330687 24953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:12.332770 24953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:12.346333 24959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:12.346333 24961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:12.346390 24958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:12.347107 24953 server_base.cc:1034] running on GCE node
I20260926 02:53:12.347774 24953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:12.348701 24953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:12.349900 24953 hybrid_clock.cc:648] HybridClock initialized: now 1790391192349857 us; error 30 us; skew 500 ppm
I20260926 02:53:12.350240 24953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:53:12.352502 24953 webserver.cc:460] Webserver started at http://127.19.185.193:35689/ using document root <none> and password file <none>
I20260926 02:53:12.353185 24953 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:12.353269 24953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:12.353586 24953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:12.356383 24953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "111512e434c6434cb46905b7a93b9d29"
format_stamp: "Formatted at 2026-09-26 02:53:12 on dist-test-slave-ktng"
server_key: "087fedd08bc937cbad0fa381f9f84749"
server_key_iv: "dae039f0e6d101e325910b46a2a90dbb"
server_key_version: "encryptionkey@0"
I20260926 02:53:12.357139 24953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "111512e434c6434cb46905b7a93b9d29"
format_stamp: "Formatted at 2026-09-26 02:53:12 on dist-test-slave-ktng"
server_key: "087fedd08bc937cbad0fa381f9f84749"
server_key_iv: "dae039f0e6d101e325910b46a2a90dbb"
server_key_version: "encryptionkey@0"
I20260926 02:53:12.362139 24953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:53:12.365578 24967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:12.366932 24953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:53:12.367100 24953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "111512e434c6434cb46905b7a93b9d29"
format_stamp: "Formatted at 2026-09-26 02:53:12 on dist-test-slave-ktng"
server_key: "087fedd08bc937cbad0fa381f9f84749"
server_key_iv: "dae039f0e6d101e325910b46a2a90dbb"
server_key_version: "encryptionkey@0"
I20260926 02:53:12.367301 24953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:12.372699 24953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:12.373425 24953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:12.373701 24953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:12.374569 24953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:12.375938 24953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:12.376050 24953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:12.376179 24953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:12.376277 24953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:12.408730 24953 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:45391
I20260926 02:53:12.408814 25080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:45391 every 8 connection(s)
I20260926 02:53:12.410084 24953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:53:12.416744 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 24953
I20260926 02:53:12.416926 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:53:12.417747 20199 external_mini_cluster.cc:1428] Setting key 2255c7faa1e31de1872589abd3d26d63
I20260926 02:53:12.419080 25083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:12.412963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:36493 (local address 127.19.185.254:46677)
0926 02:53:12.413412 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:12.413416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:12.414617 (+ 1201us) server_negotiation.cc:407] Connection header received
0926 02:53:12.416406 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:12.416412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:12.416484 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:12.416600 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:12.418220 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:12.418226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:12.418233 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:12.418256 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:12.418314 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:12.418334 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:12.418472 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:12.418699 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 02:53:12.418753 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":222,"threads_started":1}
I20260926 02:53:12.420105 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:12.413571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:46677 (local address 127.19.185.193:36493)
0926 02:53:12.414448 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:12.414489 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:12.416117 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:12.416885 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:12.416899 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:12.417017 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:12.417833 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:53:12.417837 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:12.417900 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:53:12.418008 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:12.418450 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:12.418461 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:12.418470 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 02:53:12.418748 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 02:53:12.418987 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":267,"threads_started":1}
I20260926 02:53:12.420780 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46677
--builtin_ntp_servers=127.19.185.212:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 02:53:12.424841 25081 heartbeater.cc:347] Connected to a master server at 127.19.185.254:46677
I20260926 02:53:12.425184 25081 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:12.425999 25081 heartbeater.cc:511] Master 127.19.185.254:46677 requested a full tablet report, sending...
I20260926 02:53:12.428299 24893 ts_manager.cc:194] Registered new tserver with Master: 111512e434c6434cb46905b7a93b9d29 (127.19.185.193:45391)
W20260926 02:53:12.749414 25085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:12.749682 25085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:12.749825 25085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:12.757073 25085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:12.757169 25085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:12.757335 25085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:53:12.766486 25085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:12.767908 25085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:12.769608 25085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:12.783195 25091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:12.783133 25093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:12.783393 25090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:12.784111 25085 server_base.cc:1034] running on GCE node
I20260926 02:53:12.784722 25085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:12.785727 25085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:12.786959 25085 hybrid_clock.cc:648] HybridClock initialized: now 1790391192786877 us; error 95 us; skew 500 ppm
I20260926 02:53:12.787365 25085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:53:12.789714 25085 webserver.cc:460] Webserver started at http://127.19.185.194:37121/ using document root <none> and password file <none>
I20260926 02:53:12.790427 25085 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:12.790555 25085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:12.790889 25085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:12.793841 25085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "a2cd56d2389740eaa02bfc5361806906"
format_stamp: "Formatted at 2026-09-26 02:53:12 on dist-test-slave-ktng"
server_key: "1f050fc3c5f71ce557c77d621017afce"
server_key_iv: "7f48c831c593f5a6e77ed2d1dff10ee9"
server_key_version: "encryptionkey@0"
I20260926 02:53:12.794677 25085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "a2cd56d2389740eaa02bfc5361806906"
format_stamp: "Formatted at 2026-09-26 02:53:12 on dist-test-slave-ktng"
server_key: "1f050fc3c5f71ce557c77d621017afce"
server_key_iv: "7f48c831c593f5a6e77ed2d1dff10ee9"
server_key_version: "encryptionkey@0"
I20260926 02:53:12.800201 25085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 02:53:12.803596 25099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:12.804934 25085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:53:12.805106 25085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "a2cd56d2389740eaa02bfc5361806906"
format_stamp: "Formatted at 2026-09-26 02:53:12 on dist-test-slave-ktng"
server_key: "1f050fc3c5f71ce557c77d621017afce"
server_key_iv: "7f48c831c593f5a6e77ed2d1dff10ee9"
server_key_version: "encryptionkey@0"
I20260926 02:53:12.805325 25085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:12.811106 25085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:12.812086 25085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:12.812361 25085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:12.813342 25085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:12.814805 25085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:12.814929 25085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:12.815053 25085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:12.815138 25085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:12.848451 25085 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:38681
I20260926 02:53:12.848500 25212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:38681 every 8 connection(s)
I20260926 02:53:12.849764 25085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:53:12.851676 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 25085
I20260926 02:53:12.851861 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:53:12.852447 20199 external_mini_cluster.cc:1428] Setting key 352f25e9efdd36cf7ded57483a3d85e4
I20260926 02:53:12.855499 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46677
--builtin_ntp_servers=127.19.185.212:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 02:53:12.858798 25083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:12.852446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:58989 (local address 127.19.185.254:46677)
0926 02:53:12.852670 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:12.852674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:12.854564 (+ 1890us) server_negotiation.cc:407] Connection header received
0926 02:53:12.856387 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:12.856396 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:12.856475 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:12.856585 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:12.857937 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:12.857946 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:12.857953 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:12.857975 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:12.858017 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:12.858035 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:12.858163 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:12.858368 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 02:53:12.858437 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:53:12.859635 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:12.853132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:46677 (local address 127.19.185.194:58989)
0926 02:53:12.854371 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:12.854424 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:12.856128 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:12.856772 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:12.856802 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:12.856910 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:12.857634 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:53:12.857636 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:12.857675 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:53:12.857760 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:12.858169 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:12.858176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:12.858182 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:53:12.858398 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 02:53:12.858594 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":358,"threads_started":1}
I20260926 02:53:12.861176 25213 heartbeater.cc:347] Connected to a master server at 127.19.185.254:46677
I20260926 02:53:12.861464 25213 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:12.862154 25213 heartbeater.cc:511] Master 127.19.185.254:46677 requested a full tablet report, sending...
I20260926 02:53:12.863716 24893 ts_manager.cc:194] Registered new tserver with Master: a2cd56d2389740eaa02bfc5361806906 (127.19.185.194:38681)
W20260926 02:53:13.191965 25216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:13.192240 25216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:13.192416 25216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:13.199761 25216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:13.199858 25216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:13.200070 25216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:53:13.209251 25216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:13.210647 25216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:13.212581 25216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:13.225771 25222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:13.225839 25221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:13.226307 25224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:13.226464 25216 server_base.cc:1034] running on GCE node
I20260926 02:53:13.227133 25216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:13.228093 25216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:13.229323 25216 hybrid_clock.cc:648] HybridClock initialized: now 1790391193229277 us; error 43 us; skew 500 ppm
I20260926 02:53:13.229646 25216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:53:13.231938 25216 webserver.cc:460] Webserver started at http://127.19.185.195:38715/ using document root <none> and password file <none>
I20260926 02:53:13.232674 25216 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:13.232805 25216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:13.233120 25216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:13.235967 25216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "b2fb339393234341ad700f02fd74cdb6"
format_stamp: "Formatted at 2026-09-26 02:53:13 on dist-test-slave-ktng"
server_key: "ef1f48eff0f1b259f68162d9c05a0277"
server_key_iv: "857f50e14b66cfdfa35aed80c2088868"
server_key_version: "encryptionkey@0"
I20260926 02:53:13.236763 25216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "b2fb339393234341ad700f02fd74cdb6"
format_stamp: "Formatted at 2026-09-26 02:53:13 on dist-test-slave-ktng"
server_key: "ef1f48eff0f1b259f68162d9c05a0277"
server_key_iv: "857f50e14b66cfdfa35aed80c2088868"
server_key_version: "encryptionkey@0"
I20260926 02:53:13.241765 25216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 02:53:13.245088 25230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:13.246381 25216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:53:13.246549 25216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "b2fb339393234341ad700f02fd74cdb6"
format_stamp: "Formatted at 2026-09-26 02:53:13 on dist-test-slave-ktng"
server_key: "ef1f48eff0f1b259f68162d9c05a0277"
server_key_iv: "857f50e14b66cfdfa35aed80c2088868"
server_key_version: "encryptionkey@0"
I20260926 02:53:13.246757 25216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:13.252031 25216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:13.252789 25216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:13.253049 25216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:13.253927 25216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:13.255246 25216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:13.255386 25216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:13.255512 25216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:13.255599 25216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:13.287187 25216 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:44345
I20260926 02:53:13.287264 25343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:44345 every 8 connection(s)
I20260926 02:53:13.288640 25216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:53:13.297286 25083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.291530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:59143 (local address 127.19.185.254:46677)
0926 02:53:13.291769 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:13.291775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:13.293275 (+ 1500us) server_negotiation.cc:407] Connection header received
0926 02:53:13.294915 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:13.294919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:13.294974 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:13.295064 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:13.296412 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:13.296416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:13.296419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:13.296435 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:13.296465 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:13.296476 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:13.296584 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:13.296856 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 02:53:13.296938 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:53:13.297267 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 25216
I20260926 02:53:13.297446 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:53:13.298038 20199 external_mini_cluster.cc:1428] Setting key c53562c5dadb9873dcab48f3ea70285d
I20260926 02:53:13.298043 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.292210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:46677 (local address 127.19.185.195:59143)
0926 02:53:13.293112 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:13.293156 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:13.294660 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:13.295344 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:13.295358 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:13.295472 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:13.296086 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:53:13.296089 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:13.296157 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:53:13.296243 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:13.296561 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:13.296570 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:13.296577 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 02:53:13.296809 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 02:53:13.297008 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":272,"threads_started":1}
I20260926 02:53:13.299806 25344 heartbeater.cc:347] Connected to a master server at 127.19.185.254:46677
I20260926 02:53:13.300169 25344 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:13.300876 25344 heartbeater.cc:511] Master 127.19.185.254:46677 requested a full tablet report, sending...
I20260926 02:53:13.302291 24892 ts_manager.cc:194] Registered new tserver with Master: b2fb339393234341ad700f02fd74cdb6 (127.19.185.195:44345)
I20260926 02:53:13.313447 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:53:13.320394 25083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.317851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57990 (local address 127.19.185.254:46677)
0926 02:53:13.318086 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:13.318096 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:13.318483 (+ 387us) server_negotiation.cc:407] Connection header received
0926 02:53:13.318632 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:13.318636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:13.318695 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:13.318774 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:13.319694 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:13.319698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:13.319701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:13.319716 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:13.319744 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:13.319755 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:13.319852 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:13.320046 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 02:53:13.320095 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 02:53:13.324803 24892 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:53:13.331825 24892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:53:13.352455 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.344347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:45391 (local address 127.0.0.1:44032)
0926 02:53:13.345466 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:13.345531 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:13.345713 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:13.347542 (+ 1829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:13.347552 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:13.347685 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:13.349342 (+ 1657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:53:13.349348 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:13.349385 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:53:13.350810 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:13.351458 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:13.351473 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:13.351485 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:53:13.351671 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 02:53:13.351780 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":1651,"threads_started":1}
I20260926 02:53:13.353215 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.345278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.19.185.194:38681)
0926 02:53:13.346735 (+ 1457us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:13.346742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:13.347873 (+ 1131us) server_negotiation.cc:407] Connection header received
0926 02:53:13.347970 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:13.347980 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:13.348663 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:13.348839 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:13.351529 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:13.351540 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:13.351557 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:13.351584 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:13.351646 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:13.351670 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:13.351864 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:13.352575 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 02:53:13.352699 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1309,"thread_start_us":1165,"threads_started":1}
I20260926 02:53:13.353850 25356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.344464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:38681 (local address 127.0.0.1:58986)
0926 02:53:13.347574 (+ 3110us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:13.347601 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:13.347721 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:13.349902 (+ 2181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:13.349910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:13.349944 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:13.350848 (+ 904us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:53:13.350854 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:13.350941 (+ 87us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:53:13.351003 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:13.351929 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:13.351937 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:13.351945 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 02:53:13.352144 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 02:53:13.352212 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2935,"thread_start_us":1049,"threads_started":1}
I20260926 02:53:13.355376 25357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.344513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.19.185.193:45391)
0926 02:53:13.346456 (+ 1943us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:13.346468 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:13.346501 (+ 33us) server_negotiation.cc:407] Connection header received
0926 02:53:13.346579 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:13.346586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:13.346820 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:13.346980 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:13.351055 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:13.351068 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:13.351088 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:13.351113 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:13.351190 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:13.351222 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:13.354681 (+ 3459us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:13.354773 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 02:53:13.354899 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1777,"thread_start_us":1671,"threads_started":1}
I20260926 02:53:13.358675 25148 tablet_service.cc:1467] Processing CreateTablet for tablet 33b22485ad624de8a0a1f16857d567c6 (DEFAULT_TABLE table=test-table [id=55ebe0ab32d64a2caca650a7b1981029]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:13.359546 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.346861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:44345 (local address 127.0.0.1:51218)
0926 02:53:13.353044 (+ 6183us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:13.353067 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:13.353161 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:13.357412 (+ 4251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:13.357417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:13.357450 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:13.358020 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:53:13.358023 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:13.358040 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:53:13.358084 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:13.359091 (+ 1007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:13.359097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:13.359102 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 02:53:13.359202 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 02:53:13.359257 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6038,"thread_start_us":9987,"threads_started":1}
I20260926 02:53:13.360178 25148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b22485ad624de8a0a1f16857d567c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:13.360463 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.346734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.19.185.195:44345)
0926 02:53:13.356851 (+ 10117us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:13.356862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:13.356898 (+ 36us) server_negotiation.cc:407] Connection header received
0926 02:53:13.356984 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:13.356994 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:13.357232 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:13.357396 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:13.358851 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:13.358860 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:13.358872 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:13.358888 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:13.358941 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:13.358964 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:13.359148 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:13.359209 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 02:53:13.359400 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9944,"thread_start_us":9770,"threads_started":1}
I20260926 02:53:13.360677 25015 tablet_service.cc:1467] Processing CreateTablet for tablet 33b22485ad624de8a0a1f16857d567c6 (DEFAULT_TABLE table=test-table [id=55ebe0ab32d64a2caca650a7b1981029]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:13.362023 25015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b22485ad624de8a0a1f16857d567c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:13.364028 25279 tablet_service.cc:1467] Processing CreateTablet for tablet 33b22485ad624de8a0a1f16857d567c6 (DEFAULT_TABLE table=test-table [id=55ebe0ab32d64a2caca650a7b1981029]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:13.365365 25279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b22485ad624de8a0a1f16857d567c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:13.372574 25362 tablet_bootstrap.cc:492] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906: Bootstrap starting.
I20260926 02:53:13.375610 25362 tablet_bootstrap.cc:654] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:13.376621 25363 tablet_bootstrap.cc:492] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29: Bootstrap starting.
I20260926 02:53:13.376884 25362 log.cc:824] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:13.380472 25363 tablet_bootstrap.cc:654] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:13.380939 25362 tablet_bootstrap.cc:492] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906: No bootstrap required, opened a new log
I20260926 02:53:13.381206 25362 ts_tablet_manager.cc:1397] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260926 02:53:13.381359 25365 tablet_bootstrap.cc:492] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6: Bootstrap starting.
I20260926 02:53:13.381703 25363 log.cc:824] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:13.384438 25365 tablet_bootstrap.cc:654] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:13.384730 25363 tablet_bootstrap.cc:492] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29: No bootstrap required, opened a new log
I20260926 02:53:13.384999 25363 ts_tablet_manager.cc:1397] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:53:13.385440 25365 log.cc:824] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:13.388371 25365 tablet_bootstrap.cc:492] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6: No bootstrap required, opened a new log
I20260926 02:53:13.388634 25365 ts_tablet_manager.cc:1397] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 02:53:13.391472 25362 raft_consensus.cc:348] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111512e434c6434cb46905b7a93b9d29" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45391 } } peers { permanent_uuid: "a2cd56d2389740eaa02bfc5361806906" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38681 } } peers { permanent_uuid: "b2fb339393234341ad700f02fd74cdb6" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 44345 } }
I20260926 02:53:13.391934 25362 raft_consensus.cc:374] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:13.392035 25362 raft_consensus.cc:729] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cd56d2389740eaa02bfc5361806906, State: Initialized, Role: FOLLOWER
I20260926 02:53:13.392666 25362 consensus_queue.cc:260] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111512e434c6434cb46905b7a93b9d29" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45391 } } peers { permanent_uuid: "a2cd56d2389740eaa02bfc5361806906" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38681 } } peers { permanent_uuid: "b2fb339393234341ad700f02fd74cdb6" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 44345 } }
I20260926 02:53:13.393725 25213 heartbeater.cc:503] Master 127.19.185.254:46677 was elected leader, sending a full tablet report...
I20260926 02:53:13.394356 25362 ts_tablet_manager.cc:1428] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 02:53:13.396277 25363 raft_consensus.cc:348] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111512e434c6434cb46905b7a93b9d29" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45391 } } peers { permanent_uuid: "a2cd56d2389740eaa02bfc5361806906" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38681 } } peers { permanent_uuid: "b2fb339393234341ad700f02fd74cdb6" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 44345 } }
I20260926 02:53:13.396637 25363 raft_consensus.cc:374] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:13.396407 25365 raft_consensus.cc:348] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111512e434c6434cb46905b7a93b9d29" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45391 } } peers { permanent_uuid: "a2cd56d2389740eaa02bfc5361806906" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38681 } } peers { permanent_uuid: "b2fb339393234341ad700f02fd74cdb6" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 44345 } }
I20260926 02:53:13.396756 25363 raft_consensus.cc:729] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111512e434c6434cb46905b7a93b9d29, State: Initialized, Role: FOLLOWER
I20260926 02:53:13.396824 25365 raft_consensus.cc:374] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:13.396951 25365 raft_consensus.cc:729] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2fb339393234341ad700f02fd74cdb6, State: Initialized, Role: FOLLOWER
I20260926 02:53:13.397518 25363 consensus_queue.cc:260] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111512e434c6434cb46905b7a93b9d29" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45391 } } peers { permanent_uuid: "a2cd56d2389740eaa02bfc5361806906" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38681 } } peers { permanent_uuid: "b2fb339393234341ad700f02fd74cdb6" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 44345 } }
I20260926 02:53:13.397550 25365 consensus_queue.cc:260] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111512e434c6434cb46905b7a93b9d29" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45391 } } peers { permanent_uuid: "a2cd56d2389740eaa02bfc5361806906" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38681 } } peers { permanent_uuid: "b2fb339393234341ad700f02fd74cdb6" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 44345 } }
I20260926 02:53:13.399959 25344 heartbeater.cc:503] Master 127.19.185.254:46677 was elected leader, sending a full tablet report...
I20260926 02:53:13.400583 25365 ts_tablet_manager.cc:1428] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260926 02:53:13.401294 25081 heartbeater.cc:503] Master 127.19.185.254:46677 was elected leader, sending a full tablet report...
I20260926 02:53:13.401925 25363 ts_tablet_manager.cc:1428] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
W20260926 02:53:13.412477 25082 tablet.cc:2369] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:53:13.540488 25345 tablet.cc:2369] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:53:13.601818 25214 tablet.cc:2369] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:53:13.642489 25370 raft_consensus.cc:482] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:53:13.642846 25370 raft_consensus.cc:504] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111512e434c6434cb46905b7a93b9d29" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45391 } } peers { permanent_uuid: "a2cd56d2389740eaa02bfc5361806906" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38681 } } peers { permanent_uuid: "b2fb339393234341ad700f02fd74cdb6" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 44345 } }
I20260926 02:53:13.644554 25370 leader_election.cc:288] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 111512e434c6434cb46905b7a93b9d29 (127.19.185.193:45391), a2cd56d2389740eaa02bfc5361806906 (127.19.185.194:38681)
I20260926 02:53:13.647580 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.644992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:45391 (local address 127.19.185.195:51265)
0926 02:53:13.645307 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:13.645327 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:13.645455 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:13.645853 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:13.645858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:13.645887 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:13.646574 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:53:13.646579 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:13.646600 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:53:13.646656 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:13.647013 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:13.647020 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:13.647025 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 02:53:13.647209 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 02:53:13.647259 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260926 02:53:13.647866 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.645127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:45735 (local address 127.19.185.194:38681)
0926 02:53:13.645475 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:13.645484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:13.645921 (+ 437us) server_negotiation.cc:407] Connection header received
0926 02:53:13.646064 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:13.646068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:13.646112 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:13.646274 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:13.647250 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:13.647253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:13.647256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:13.647271 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:13.647349 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:13.647362 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:13.647456 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:13.647581 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 02:53:13.647631 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260926 02:53:13.647853 25357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.645114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:51265 (local address 127.19.185.193:45391)
0926 02:53:13.645300 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:13.645306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:13.645544 (+ 238us) server_negotiation.cc:407] Connection header received
0926 02:53:13.645620 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:13.645626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:13.645691 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:13.645797 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:13.646805 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:13.646812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:13.646819 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:13.646841 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:13.646882 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:13.646900 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:13.647014 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:13.647530 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 02:53:13.647592 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:53:13.648092 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.644992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:38681 (local address 127.19.185.195:45735)
0926 02:53:13.645842 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:13.645858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:13.645955 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:13.646299 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:13.646303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:13.646320 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:13.647045 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:53:13.647048 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:13.647063 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:53:13.647100 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:13.647450 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:13.647455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:13.647460 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 02:53:13.647617 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 02:53:13.647666 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":37,"thread_start_us":418,"threads_started":1}
I20260926 02:53:13.648823 25036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b22485ad624de8a0a1f16857d567c6" candidate_uuid: "b2fb339393234341ad700f02fd74cdb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111512e434c6434cb46905b7a93b9d29" is_pre_election: true
I20260926 02:53:13.649022 25168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b22485ad624de8a0a1f16857d567c6" candidate_uuid: "b2fb339393234341ad700f02fd74cdb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cd56d2389740eaa02bfc5361806906" is_pre_election: true
I20260926 02:53:13.649353 25036 raft_consensus.cc:2454] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2fb339393234341ad700f02fd74cdb6 in term 0.
I20260926 02:53:13.649540 25168 raft_consensus.cc:2454] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2fb339393234341ad700f02fd74cdb6 in term 0.
I20260926 02:53:13.650116 25233 leader_election.cc:302] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 111512e434c6434cb46905b7a93b9d29, a2cd56d2389740eaa02bfc5361806906, b2fb339393234341ad700f02fd74cdb6; no voters:
I20260926 02:53:13.650527 25370 raft_consensus.cc:2781] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:53:13.650681 25370 raft_consensus.cc:482] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:53:13.650784 25370 raft_consensus.cc:3037] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:13.653064 25370 raft_consensus.cc:504] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111512e434c6434cb46905b7a93b9d29" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45391 } } peers { permanent_uuid: "a2cd56d2389740eaa02bfc5361806906" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38681 } } peers { permanent_uuid: "b2fb339393234341ad700f02fd74cdb6" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 44345 } }
I20260926 02:53:13.653640 25370 leader_election.cc:288] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [CANDIDATE]: Term 1 election: Requested vote from peers 111512e434c6434cb46905b7a93b9d29 (127.19.185.193:45391), a2cd56d2389740eaa02bfc5361806906 (127.19.185.194:38681)
I20260926 02:53:13.653976 25036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b22485ad624de8a0a1f16857d567c6" candidate_uuid: "b2fb339393234341ad700f02fd74cdb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111512e434c6434cb46905b7a93b9d29"
I20260926 02:53:13.654090 25168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b22485ad624de8a0a1f16857d567c6" candidate_uuid: "b2fb339393234341ad700f02fd74cdb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cd56d2389740eaa02bfc5361806906"
I20260926 02:53:13.654172 25036 raft_consensus.cc:3037] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:13.654317 25168 raft_consensus.cc:3037] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:13.657303 25036 raft_consensus.cc:2454] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2fb339393234341ad700f02fd74cdb6 in term 1.
I20260926 02:53:13.657758 25233 leader_election.cc:302] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111512e434c6434cb46905b7a93b9d29, b2fb339393234341ad700f02fd74cdb6; no voters:
I20260926 02:53:13.658107 25370 raft_consensus.cc:2781] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:13.658648 25370 raft_consensus.cc:686] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [term 1 LEADER]: Becoming Leader. State: Replica: b2fb339393234341ad700f02fd74cdb6, State: Running, Role: LEADER
I20260926 02:53:13.659526 25370 consensus_queue.cc:237] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111512e434c6434cb46905b7a93b9d29" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45391 } } peers { permanent_uuid: "a2cd56d2389740eaa02bfc5361806906" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38681 } } peers { permanent_uuid: "b2fb339393234341ad700f02fd74cdb6" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 44345 } }
I20260926 02:53:13.660072 25168 raft_consensus.cc:2454] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2fb339393234341ad700f02fd74cdb6 in term 1.
I20260926 02:53:13.665230 24892 catalog_manager.cc:5495] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2fb339393234341ad700f02fd74cdb6 (127.19.185.195). New cstate: current_term: 1 leader_uuid: "b2fb339393234341ad700f02fd74cdb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111512e434c6434cb46905b7a93b9d29" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2cd56d2389740eaa02bfc5361806906" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2fb339393234341ad700f02fd74cdb6" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 44345 } health_report { overall_health: HEALTHY } } }
I20260926 02:53:13.723449 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.721062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.19.185.195:44345)
0926 02:53:13.721267 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:13.721273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:13.721407 (+ 134us) server_negotiation.cc:407] Connection header received
0926 02:53:13.721676 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:13.721680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:13.721759 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:13.721880 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:13.722796 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:13.722801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:13.722806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:13.722829 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:13.722866 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:13.722878 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:13.723025 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:13.723109 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 02:53:13.723167 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:53:13.731998 25168 raft_consensus.cc:1261] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906 [term 1 FOLLOWER]: Refusing update from remote peer b2fb339393234341ad700f02fd74cdb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:13.732024 25036 raft_consensus.cc:1261] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29 [term 1 FOLLOWER]: Refusing update from remote peer b2fb339393234341ad700f02fd74cdb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:13.733035 25372 consensus_queue.cc:1035] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2cd56d2389740eaa02bfc5361806906" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:13.733449 25370 consensus_queue.cc:1035] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "111512e434c6434cb46905b7a93b9d29" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:13.758596 24892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:13.758884 24892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:13.762514 24892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7c342a388d84ce3834c34d94feb6d12: Sending DeleteTablet for 3 replicas of tablet 33b22485ad624de8a0a1f16857d567c6
I20260926 02:53:13.763589 25015 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b22485ad624de8a0a1f16857d567c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:13 UTC) from {username='slave'} at 127.0.0.1:44032
I20260926 02:53:13.763706 25279 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b22485ad624de8a0a1f16857d567c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:13 UTC) from {username='slave'} at 127.0.0.1:51218
I20260926 02:53:13.763695 25148 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b22485ad624de8a0a1f16857d567c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:13 UTC) from {username='slave'} at 127.0.0.1:58986
I20260926 02:53:13.767714 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 24953
I20260926 02:53:13.771216 25386 tablet_replica.cc:330] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6: stopping tablet replica
I20260926 02:53:13.775218 25357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.769855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.19.185.193:45391)
0926 02:53:13.770102 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:13.770111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:13.772428 (+ 2317us) server_negotiation.cc:407] Connection header received
0926 02:53:13.772575 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:13.772585 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:13.772719 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:13.772848 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:13.774197 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:13.774206 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:13.774214 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:13.774240 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:13.774289 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:13.774311 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:13.774426 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:13.774693 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 02:53:13.774768 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:53:13.775238 25387 tablet_replica.cc:330] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906: stopping tablet replica
I20260926 02:53:13.775856 25387 raft_consensus.cc:2229] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:13.776400 25387 raft_consensus.cc:2258] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:13.776401 25076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:13.778792 25387 ts_tablet_manager.cc:1905] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:13.783434 25386 raft_consensus.cc:2229] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:53:13.784067 25386 raft_consensus.cc:2258] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:13.785631 25387 ts_tablet_manager.cc:1918] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:13.785784 25387 log.cc:1196] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/33b22485ad624de8a0a1f16857d567c6
I20260926 02:53:13.786051 25386 ts_tablet_manager.cc:1905] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:13.786302 25387 ts_tablet_manager.cc:1939] T 33b22485ad624de8a0a1f16857d567c6 P a2cd56d2389740eaa02bfc5361806906: Deleting consensus metadata
I20260926 02:53:13.788069 24877 catalog_manager.cc:4841] TS a2cd56d2389740eaa02bfc5361806906 (127.19.185.194:38681): tablet 33b22485ad624de8a0a1f16857d567c6 (table test-table [id=55ebe0ab32d64a2caca650a7b1981029]) successfully deleted
I20260926 02:53:13.794119 25386 ts_tablet_manager.cc:1918] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:13.794313 25386 log.cc:1196] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/33b22485ad624de8a0a1f16857d567c6
I20260926 02:53:13.794777 25386 ts_tablet_manager.cc:1939] T 33b22485ad624de8a0a1f16857d567c6 P b2fb339393234341ad700f02fd74cdb6: Deleting consensus metadata
I20260926 02:53:13.796319 24879 catalog_manager.cc:4841] TS b2fb339393234341ad700f02fd74cdb6 (127.19.185.195:44345): tablet 33b22485ad624de8a0a1f16857d567c6 (table test-table [id=55ebe0ab32d64a2caca650a7b1981029]) successfully deleted
I20260926 02:53:13.924569 25385 tablet_replica.cc:330] T 33b22485ad624de8a0a1f16857d567c6 P 111512e434c6434cb46905b7a93b9d29: stopping tablet replica
I20260926 02:53:13.924898 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 24953
W20260926 02:53:13.943437 24879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.185.193:45391 (error 108)
I20260926 02:53:13.944169 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 25085
I20260926 02:53:13.945948 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.944726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:45391 (local address 127.0.0.1:44046)
0926 02:53:13.945583 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:13.945751 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":660,"threads_started":1}
W20260926 02:53:13.946267 24879 catalog_manager.cc:4584] TS 111512e434c6434cb46905b7a93b9d29 (127.19.185.193:45391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33b22485ad624de8a0a1f16857d567c6: Network error: Client connection negotiation failed: client connection to 127.19.185.193:45391: connect: Connection refused (error 111)
I20260926 02:53:13.947422 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.944945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.19.185.194:38681)
0926 02:53:13.945183 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:13.945192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:13.945218 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:53:13.945407 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:13.945413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:13.945531 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:13.945633 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:13.946663 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:13.946669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:13.946674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:13.946694 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:13.946733 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:13.946748 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:13.946878 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:13.946947 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 02:53:13.947008 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:53:13.948089 25208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:14.000168 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:13.999632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:45391 (local address 127.0.0.1:44052)
0926 02:53:13.999867 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:13.999985 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 02:53:14.001344 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:14.000942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:45391 (local address 127.0.0.1:44062)
0926 02:53:14.001105 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:14.001198 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 02:53:14.038287 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 25085
I20260926 02:53:14.055413 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 25216
I20260926 02:53:14.058681 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:14.056312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.19.185.195:44345)
0926 02:53:14.056537 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:14.056545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:14.056595 (+ 50us) server_negotiation.cc:407] Connection header received
0926 02:53:14.056686 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:14.056692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:14.056764 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:14.056884 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:14.057955 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:14.057961 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:14.057966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:14.057985 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:14.058022 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:14.058041 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:14.058193 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:14.058256 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 02:53:14.058325 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:53:14.059461 25339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:14.061822 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:14.061314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:45391 (local address 127.0.0.1:44064)
0926 02:53:14.061616 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:14.061700 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260926 02:53:14.062983 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:14.062631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:45391 (local address 127.0.0.1:44080)
0926 02:53:14.062786 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:14.062860 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 02:53:14.143790 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:14.143219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:45391 (local address 127.0.0.1:44082)
0926 02:53:14.143545 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:14.143636 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260926 02:53:14.144932 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:14.144537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:45391 (local address 127.0.0.1:44084)
0926 02:53:14.144730 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:14.144803 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 02:53:14.179899 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 25216
I20260926 02:53:14.194867 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 24862
I20260926 02:53:14.196019 24924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:14.300040 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 24862
2026-09-26T02:53:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2884 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 02:53:14.341339 20199 mini_oidc.cc:152] Starting JWKS server
I20260926 02:53:14.344050 20199 webserver.cc:460] Webserver started at https://127.0.0.1:36031/ using document root <none> and password file <none>
I20260926 02:53:14.344368 20199 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 02:53:14.345026 20199 webserver.cc:460] Webserver started at http://127.0.0.1:38541/ using document root <none> and password file <none>
2026-09-26T02:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:53:14Z Disabled control of system clock
I20260926 02:53:14.371517 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:43367
--webserver_interface=127.19.185.254
--webserver_port=0
--builtin_ntp_servers=127.19.185.212:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:43367
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36031/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/testchainca.cert with env {}
W20260926 02:53:14.690219 25401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:14.690500 25401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:14.690585 25401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:14.697817 25401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:14.697943 25401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:14.697981 25401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:14.698030 25401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:14.698076 25401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:53:14.698141 25401 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:53:14.698180 25401 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36031/jwks.json
I20260926 02:53:14.708012 25401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:43367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:43367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36031/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:14.709239 25401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:14.710485 25401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:14.722594 25406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:14.722636 25407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:14.722661 25409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:14.724007 25401 server_base.cc:1034] running on GCE node
I20260926 02:53:14.724797 25401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:14.726202 25401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:14.727427 25401 hybrid_clock.cc:648] HybridClock initialized: now 1790391194727375 us; error 30 us; skew 500 ppm
I20260926 02:53:14.727818 25401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:53:14.730124 25401 webserver.cc:460] Webserver started at http://127.19.185.254:36021/ using document root <none> and password file <none>
I20260926 02:53:14.730810 25401 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:14.730927 25401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:14.731251 25401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:14.734084 25401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "efd5484ec0044f89b148ec6c1cf25f72"
format_stamp: "Formatted at 2026-09-26 02:53:14 on dist-test-slave-ktng"
server_key: "635974f605d1ab3cd87b8b94579cd84a"
server_key_iv: "dabbfdd10b36d813ecace3c6dbe1e689"
server_key_version: "encryptionkey@0"
I20260926 02:53:14.734858 25401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "efd5484ec0044f89b148ec6c1cf25f72"
format_stamp: "Formatted at 2026-09-26 02:53:14 on dist-test-slave-ktng"
server_key: "635974f605d1ab3cd87b8b94579cd84a"
server_key_iv: "dabbfdd10b36d813ecace3c6dbe1e689"
server_key_version: "encryptionkey@0"
I20260926 02:53:14.740087 25401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 02:53:14.743552 25415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:14.744892 25401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:53:14.745076 25401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "efd5484ec0044f89b148ec6c1cf25f72"
format_stamp: "Formatted at 2026-09-26 02:53:14 on dist-test-slave-ktng"
server_key: "635974f605d1ab3cd87b8b94579cd84a"
server_key_iv: "dabbfdd10b36d813ecace3c6dbe1e689"
server_key_version: "encryptionkey@0"
I20260926 02:53:14.745322 25401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:14.763286 25401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:14.764178 25401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:14.764500 25401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:14.782879 25401 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:43367
I20260926 02:53:14.782886 25466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:43367 every 8 connection(s)
I20260926 02:53:14.784209 25401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:53:14.788082 25467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:14.788898 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 25401
I20260926 02:53:14.789206 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:53:14.789839 20199 external_mini_cluster.cc:1428] Setting key 49735edc2ffb8116f251a1be7db6f260
I20260926 02:53:14.801715 25467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72: Bootstrap starting.
I20260926 02:53:14.803696 25470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:14.791932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.19.185.254:43367)
0926 02:53:14.792971 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:14.792984 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:14.793024 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:53:14.794558 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:14.794577 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:14.795028 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:14.795405 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:14.795968 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:14.797204 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:14.799095 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:14.799915 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:14.800056 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:14.801331 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:14.801338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:14.801356 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:14.801920 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:14.802026 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:14.802047 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:14.802134 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:14.802358 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 02:53:14.802707 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":403,"threads_started":1}
I20260926 02:53:14.805292 25467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:14.806416 25467 log.cc:824] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:14.807560 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:53:14.809269 25467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72: No bootstrap required, opened a new log
I20260926 02:53:14.816926 25467 raft_consensus.cc:348] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd5484ec0044f89b148ec6c1cf25f72" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43367 } }
I20260926 02:53:14.817303 25467 raft_consensus.cc:374] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:14.817413 25467 raft_consensus.cc:729] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd5484ec0044f89b148ec6c1cf25f72, State: Initialized, Role: FOLLOWER
I20260926 02:53:14.818127 25467 consensus_queue.cc:260] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd5484ec0044f89b148ec6c1cf25f72" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43367 } }
I20260926 02:53:14.818396 25467 raft_consensus.cc:388] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:53:14.818552 25467 raft_consensus.cc:482] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:53:14.818732 25467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:14.820905 25467 raft_consensus.cc:504] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd5484ec0044f89b148ec6c1cf25f72" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43367 } }
I20260926 02:53:14.821405 25467 leader_election.cc:302] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efd5484ec0044f89b148ec6c1cf25f72; no voters:
I20260926 02:53:14.821950 25467 leader_election.cc:288] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:53:14.822104 25472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:14.822824 25472 raft_consensus.cc:686] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [term 1 LEADER]: Becoming Leader. State: Replica: efd5484ec0044f89b148ec6c1cf25f72, State: Running, Role: LEADER
I20260926 02:53:14.823421 25472 consensus_queue.cc:237] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd5484ec0044f89b148ec6c1cf25f72" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43367 } }
I20260926 02:53:14.824079 25467 sys_catalog.cc:564] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:14.826299 25474 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efd5484ec0044f89b148ec6c1cf25f72. Latest consensus state: current_term: 1 leader_uuid: "efd5484ec0044f89b148ec6c1cf25f72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd5484ec0044f89b148ec6c1cf25f72" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43367 } } }
I20260926 02:53:14.826419 25473 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efd5484ec0044f89b148ec6c1cf25f72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd5484ec0044f89b148ec6c1cf25f72" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43367 } } }
I20260926 02:53:14.826931 25474 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:14.826973 25473 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:14.828857 25479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:53:14.834339 25479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:53:14.843502 25479 catalog_manager.cc:1347] Generated new cluster ID: fc9cac07cadf4728a668b08e7d7b8886
I20260926 02:53:14.843616 25479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:53:14.856896 25479 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:53:14.858285 25479 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:53:14.866475 25479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efd5484ec0044f89b148ec6c1cf25f72: Generated new TSK 0
I20260926 02:53:14.867352 25479 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 02:53:14.908677 25497 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 02:53:14.908855 25497 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 02:53:14.910009 25470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:14.888740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54074 (local address 127.19.185.254:43367)
0926 02:53:14.888936 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:14.888943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:14.889361 (+ 418us) server_negotiation.cc:407] Connection header received
0926 02:53:14.889642 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:14.889647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:14.889755 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:14.889907 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:53:14.890406 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:14.891560 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:14.892940 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:14.893640 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:14.893767 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:14.894255 (+ 488us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:53:14.909332 (+ 15077us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 02:53:14.909706 (+ 374us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54074: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 02:53:14.910141 25470 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54074: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 02:53:14.911401 25496 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:14.888602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43367 (local address 127.0.0.1:54074)
0926 02:53:14.889256 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:14.889272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:14.889418 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:14.889930 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:14.889933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:14.889943 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:53:14.890242 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:14.890250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:14.891690 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:14.891694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:14.892798 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:14.892810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:14.893809 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:14.893813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:14.893933 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:14.893989 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:53:14.909622 (+ 15633us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 02:53:14.909875 (+ 253us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.185.254:43367: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":344,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.19.185.254:43367: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 02:53:14.913436 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 25401
I20260926 02:53:14.914553 25462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:15.006388 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 25401
2026-09-26T02:53:15Z chronyd exiting
I20260926 02:53:15.056640 20199 test_util.cc:175] -----------------------------------------------
I20260926 02:53:15.056815 20199 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391151502216-20199-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (719 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 02:53:15.060537 20199 mini_oidc.cc:152] Starting JWKS server
I20260926 02:53:15.062354 20199 webserver.cc:460] Webserver started at https://127.0.0.1:33125/ using document root <none> and password file <none>
I20260926 02:53:15.062590 20199 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 02:53:15.063251 20199 webserver.cc:460] Webserver started at http://127.0.0.1:41307/ using document root <none> and password file <none>
2026-09-26T02:53:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:53:15Z Disabled control of system clock
I20260926 02:53:15.088899 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:36895
--webserver_interface=127.19.185.254
--webserver_port=0
--builtin_ntp_servers=127.19.185.212:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:36895
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33125/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/testchainca.cert with env {}
W20260926 02:53:15.406894 25504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:15.407239 25504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:15.407411 25504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:15.414700 25504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:15.414850 25504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:15.414939 25504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:15.415041 25504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:15.415122 25504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:53:15.415211 25504 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:53:15.415285 25504 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33125/jwks.json
I20260926 02:53:15.425531 25504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:36895
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:36895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33125/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:15.426899 25504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:15.428321 25504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:15.440361 25509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:15.440460 25512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:15.440410 25510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:15.441167 25504 server_base.cc:1034] running on GCE node
I20260926 02:53:15.441941 25504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:15.443450 25504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:15.444677 25504 hybrid_clock.cc:648] HybridClock initialized: now 1790391195444636 us; error 52 us; skew 500 ppm
I20260926 02:53:15.445210 25504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:53:15.447546 25504 webserver.cc:460] Webserver started at http://127.19.185.254:39217/ using document root <none> and password file <none>
I20260926 02:53:15.448221 25504 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:15.448349 25504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:15.448694 25504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:15.451519 25504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "cca1f109cdd648a99e9ec149dd20d91d"
format_stamp: "Formatted at 2026-09-26 02:53:15 on dist-test-slave-ktng"
server_key: "df3ee7e5ebaf72b0f4c6d4f23d701bad"
server_key_iv: "e830381b06752e26d720995be452bb6b"
server_key_version: "encryptionkey@0"
I20260926 02:53:15.452296 25504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "cca1f109cdd648a99e9ec149dd20d91d"
format_stamp: "Formatted at 2026-09-26 02:53:15 on dist-test-slave-ktng"
server_key: "df3ee7e5ebaf72b0f4c6d4f23d701bad"
server_key_iv: "e830381b06752e26d720995be452bb6b"
server_key_version: "encryptionkey@0"
I20260926 02:53:15.457446 25504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:53:15.460945 25518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:15.462455 25504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:53:15.462643 25504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "cca1f109cdd648a99e9ec149dd20d91d"
format_stamp: "Formatted at 2026-09-26 02:53:15 on dist-test-slave-ktng"
server_key: "df3ee7e5ebaf72b0f4c6d4f23d701bad"
server_key_iv: "e830381b06752e26d720995be452bb6b"
server_key_version: "encryptionkey@0"
I20260926 02:53:15.462873 25504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:15.486774 25504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:15.487679 25504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:15.487987 25504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:15.506364 25504 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:36895
I20260926 02:53:15.506385 25569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:36895 every 8 connection(s)
I20260926 02:53:15.507805 25504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:53:15.511541 25570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:15.516546 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 25504
I20260926 02:53:15.516767 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:53:15.517405 20199 external_mini_cluster.cc:1428] Setting key f514cdcfc185589adeecfed8175a3187
I20260926 02:53:15.522696 25570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d: Bootstrap starting.
I20260926 02:53:15.525938 25570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:15.527050 25570 log.cc:824] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:15.530638 25570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d: No bootstrap required, opened a new log
I20260926 02:53:15.531186 25573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:15.519556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.19.185.254:36895)
0926 02:53:15.520550 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:15.520561 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:15.520602 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:53:15.522246 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:15.522268 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:15.522722 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:15.523124 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:15.523711 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:15.525006 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:15.526342 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:15.527013 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:15.527213 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:15.528425 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:15.528436 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:15.528453 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:15.529166 (+ 713us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:15.529263 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:15.529290 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:15.529432 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:15.529626 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 02:53:15.529943 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":355,"threads_started":1}
I20260926 02:53:15.534895 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:53:15.539134 25570 raft_consensus.cc:348] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca1f109cdd648a99e9ec149dd20d91d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36895 } }
I20260926 02:53:15.539546 25570 raft_consensus.cc:374] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:15.539655 25570 raft_consensus.cc:729] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca1f109cdd648a99e9ec149dd20d91d, State: Initialized, Role: FOLLOWER
I20260926 02:53:15.540205 25570 consensus_queue.cc:260] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca1f109cdd648a99e9ec149dd20d91d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36895 } }
I20260926 02:53:15.540450 25570 raft_consensus.cc:388] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:53:15.540604 25570 raft_consensus.cc:482] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:53:15.540791 25570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:15.543220 25570 raft_consensus.cc:504] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca1f109cdd648a99e9ec149dd20d91d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36895 } }
I20260926 02:53:15.543769 25570 leader_election.cc:302] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cca1f109cdd648a99e9ec149dd20d91d; no voters:
I20260926 02:53:15.544378 25570 leader_election.cc:288] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:53:15.544507 25575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:15.545243 25575 raft_consensus.cc:686] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [term 1 LEADER]: Becoming Leader. State: Replica: cca1f109cdd648a99e9ec149dd20d91d, State: Running, Role: LEADER
I20260926 02:53:15.546346 25575 consensus_queue.cc:237] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca1f109cdd648a99e9ec149dd20d91d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36895 } }
I20260926 02:53:15.546622 25570 sys_catalog.cc:564] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:15.549089 25576 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cca1f109cdd648a99e9ec149dd20d91d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca1f109cdd648a99e9ec149dd20d91d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36895 } } }
I20260926 02:53:15.549566 25576 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:15.549408 25577 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cca1f109cdd648a99e9ec149dd20d91d. Latest consensus state: current_term: 1 leader_uuid: "cca1f109cdd648a99e9ec149dd20d91d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca1f109cdd648a99e9ec149dd20d91d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36895 } } }
I20260926 02:53:15.549727 25577 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:15.551857 25584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:53:15.556314 25584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:53:15.564725 25584 catalog_manager.cc:1347] Generated new cluster ID: 471d1a99d33f415d88416bcd75ba4626
I20260926 02:53:15.564842 25584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:53:15.580145 25584 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:53:15.581377 25584 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:53:15.586462 25584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cca1f109cdd648a99e9ec149dd20d91d: Generated new TSK 0
I20260926 02:53:15.587427 25584 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 02:53:15.627900 25600 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 02:53:15.628026 25600 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260926 02:53:15.629336 25599 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:15.607827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:36895 (local address 127.0.0.1:36200)
0926 02:53:15.608496 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:15.608517 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:15.608655 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:15.609131 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:15.609136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:15.609145 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:53:15.609393 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:15.609400 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:15.610811 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:15.610814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:15.611957 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:15.611969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:15.612933 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:15.612938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:15.613062 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:15.613094 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:53:15.628864 (+ 15770us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 02:53:15.629059 (+ 195us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.185.254:36895: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":392,"threads_started":1}
I20260926 02:53:15.629418 25573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:15.607970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.19.185.254:36895)
0926 02:53:15.608169 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:15.608176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:15.608615 (+ 439us) server_negotiation.cc:407] Connection header received
0926 02:53:15.608847 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:15.608853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:15.608939 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:15.609121 (+ 182us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:53:15.609535 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:15.610673 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:15.612094 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:15.612773 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:15.612913 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:15.613340 (+ 427us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:53:15.628612 (+ 15272us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 02:53:15.628998 (+ 386us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36200: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 02:53:15.629546 25573 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36200: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 02:53:15.630096 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 25504
I20260926 02:53:15.631265 25565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:15.723737 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 25504
2026-09-26T02:53:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (718 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 02:53:15.779124 20199 mini_oidc.cc:152] Starting JWKS server
I20260926 02:53:15.781339 20199 webserver.cc:460] Webserver started at https://127.0.0.1:39695/ using document root <none> and password file <none>
I20260926 02:53:15.781554 20199 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 02:53:15.782151 20199 webserver.cc:460] Webserver started at http://127.0.0.1:40529/ using document root <none> and password file <none>
2026-09-26T02:53:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:53:15Z Disabled control of system clock
I20260926 02:53:15.809352 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:41401
--webserver_interface=127.19.185.254
--webserver_port=0
--builtin_ntp_servers=127.19.185.212:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:41401
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39695/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 02:53:16.132037 25607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:16.132325 25607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:16.132411 25607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:16.140188 25607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:16.140295 25607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:16.140331 25607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:16.140389 25607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:16.140432 25607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:53:16.140486 25607 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:53:16.140527 25607 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39695/jwks.json
I20260926 02:53:16.149520 25607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:41401
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:41401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39695/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:16.150797 25607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:16.152674 25607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:16.164786 25612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:16.164826 25613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:16.165345 25615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:16.165925 25607 server_base.cc:1034] running on GCE node
I20260926 02:53:16.166810 25607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:16.168255 25607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:16.169483 25607 hybrid_clock.cc:648] HybridClock initialized: now 1790391196169437 us; error 42 us; skew 500 ppm
I20260926 02:53:16.169827 25607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:53:16.172081 25607 webserver.cc:460] Webserver started at http://127.19.185.254:43011/ using document root <none> and password file <none>
I20260926 02:53:16.172762 25607 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:16.172891 25607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:16.173216 25607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:16.176147 25607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "fecf88e32b9a4e329bb22a6535a78b3f"
format_stamp: "Formatted at 2026-09-26 02:53:16 on dist-test-slave-ktng"
server_key: "af58adb451b55ca045e222cfbb7370b5"
server_key_iv: "09ced5b7e420d6ed79b354e4fd90da86"
server_key_version: "encryptionkey@0"
I20260926 02:53:16.176998 25607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "fecf88e32b9a4e329bb22a6535a78b3f"
format_stamp: "Formatted at 2026-09-26 02:53:16 on dist-test-slave-ktng"
server_key: "af58adb451b55ca045e222cfbb7370b5"
server_key_iv: "09ced5b7e420d6ed79b354e4fd90da86"
server_key_version: "encryptionkey@0"
I20260926 02:53:16.181970 25607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:53:16.185451 25621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:16.186914 25607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:53:16.187166 25607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "fecf88e32b9a4e329bb22a6535a78b3f"
format_stamp: "Formatted at 2026-09-26 02:53:16 on dist-test-slave-ktng"
server_key: "af58adb451b55ca045e222cfbb7370b5"
server_key_iv: "09ced5b7e420d6ed79b354e4fd90da86"
server_key_version: "encryptionkey@0"
I20260926 02:53:16.187427 25607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:16.225492 25607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:16.226500 25607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:16.226861 25607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:16.245536 25607 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:41401
I20260926 02:53:16.245556 25672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:41401 every 8 connection(s)
I20260926 02:53:16.246905 25607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:53:16.248522 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 25607
I20260926 02:53:16.248740 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:53:16.249536 20199 external_mini_cluster.cc:1428] Setting key 8572879e7b9f768a6fc808e591595a9f
I20260926 02:53:16.251699 25673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:16.262353 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:16.251450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.19.185.254:41401)
0926 02:53:16.252435 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:16.252453 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:16.252494 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:53:16.254058 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:16.254079 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:16.254548 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:16.254947 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:16.255497 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:16.256820 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:16.258096 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:16.258734 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:16.258922 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:16.260181 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:16.260189 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:16.260202 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:16.260756 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:16.260856 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:16.260875 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:16.261006 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:16.261215 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 02:53:16.261495 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":372,"threads_started":1}
I20260926 02:53:16.262852 25673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f: Bootstrap starting.
I20260926 02:53:16.266000 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:53:16.266094 25673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:16.267067 25673 log.cc:824] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:16.270062 25673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f: No bootstrap required, opened a new log
I20260926 02:53:16.277711 25673 raft_consensus.cc:348] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fecf88e32b9a4e329bb22a6535a78b3f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41401 } }
I20260926 02:53:16.278091 25673 raft_consensus.cc:374] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:16.278210 25673 raft_consensus.cc:729] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fecf88e32b9a4e329bb22a6535a78b3f, State: Initialized, Role: FOLLOWER
I20260926 02:53:16.278753 25673 consensus_queue.cc:260] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fecf88e32b9a4e329bb22a6535a78b3f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41401 } }
I20260926 02:53:16.279035 25673 raft_consensus.cc:388] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:53:16.279170 25673 raft_consensus.cc:482] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:53:16.279371 25673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:16.281821 25673 raft_consensus.cc:504] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fecf88e32b9a4e329bb22a6535a78b3f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41401 } }
I20260926 02:53:16.282353 25673 leader_election.cc:302] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fecf88e32b9a4e329bb22a6535a78b3f; no voters:
I20260926 02:53:16.282891 25673 leader_election.cc:288] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:53:16.283011 25678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:16.283730 25678 raft_consensus.cc:686] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [term 1 LEADER]: Becoming Leader. State: Replica: fecf88e32b9a4e329bb22a6535a78b3f, State: Running, Role: LEADER
I20260926 02:53:16.284204 25678 consensus_queue.cc:237] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fecf88e32b9a4e329bb22a6535a78b3f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41401 } }
I20260926 02:53:16.284911 25673 sys_catalog.cc:564] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:16.286926 25679 sys_catalog.cc:455] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fecf88e32b9a4e329bb22a6535a78b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fecf88e32b9a4e329bb22a6535a78b3f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41401 } } }
I20260926 02:53:16.287511 25680 sys_catalog.cc:455] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fecf88e32b9a4e329bb22a6535a78b3f. Latest consensus state: current_term: 1 leader_uuid: "fecf88e32b9a4e329bb22a6535a78b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fecf88e32b9a4e329bb22a6535a78b3f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41401 } } }
I20260926 02:53:16.287847 25680 sys_catalog.cc:458] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:16.287847 25679 sys_catalog.cc:458] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:16.288662 25684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:53:16.293736 25684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:53:16.302948 25684 catalog_manager.cc:1347] Generated new cluster ID: a0194fcb560d42eebc43c0d898f0b1b4
I20260926 02:53:16.303066 25684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:53:16.330226 25684 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:53:16.332058 25684 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:53:16.339355 25684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fecf88e32b9a4e329bb22a6535a78b3f: Generated new TSK 0
I20260926 02:53:16.340188 25684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:53:16.375021 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:16.366962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46344 (local address 127.19.185.254:41401)
0926 02:53:16.367152 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:16.367159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:16.367547 (+ 388us) server_negotiation.cc:407] Connection header received
0926 02:53:16.367762 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:16.367772 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:16.367868 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:16.368029 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:53:16.368446 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:16.369519 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:16.370844 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:16.371568 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:16.371698 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:16.372145 (+ 447us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:53:16.374379 (+ 2234us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 02:53:16.374725 (+ 346us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46344: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 02:53:16.375172 25676 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46344: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 02:53:16.376757 25702 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:16.366843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:41401 (local address 127.0.0.1:46344)
0926 02:53:16.367455 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:16.367470 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:16.367581 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:16.368043 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:16.368046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:16.368056 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:53:16.368312 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:16.368320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:16.369659 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:16.369662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:16.370713 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:16.370724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:16.371751 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:16.371755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:16.371870 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:16.371899 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:53:16.374580 (+ 2681us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 02:53:16.376359 (+ 1779us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.185.254:41401: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":1510,"thread_start_us":336,"threads_started":1}
I20260926 02:53:16.377737 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 25607
I20260926 02:53:16.378813 25668 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 02:53:16.390458 25703 webserver.cc:533] Webserver: sslize failed:
E20260926 02:53:16.390602 25703 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 02:53:16.473801 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 25607
2026-09-26T02:53:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (749 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T02:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:53:16Z Disabled control of system clock
I20260926 02:53:16.555913 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:43971
--webserver_interface=127.19.185.254
--webserver_port=0
--builtin_ntp_servers=127.19.185.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:43971
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/testchainca.cert with env {}
W20260926 02:53:16.871014 25708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:16.871361 25708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:16.871523 25708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:16.878677 25708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:16.878803 25708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:16.878908 25708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:16.879020 25708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:16.879105 25708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:53:16.879194 25708 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:53:16.879252 25708 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 02:53:16.889106 25708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:43971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:43971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:16.890470 25708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:16.891819 25708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:16.903684 25713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:16.903786 25714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:16.903653 25716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:16.904407 25708 server_base.cc:1034] running on GCE node
I20260926 02:53:16.905170 25708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:16.906509 25708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:16.907729 25708 hybrid_clock.cc:648] HybridClock initialized: now 1790391196907687 us; error 54 us; skew 500 ppm
I20260926 02:53:16.908071 25708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:53:16.915539 25708 webserver.cc:460] Webserver started at http://127.19.185.254:43567/ using document root <none> and password file <none>
I20260926 02:53:16.916280 25708 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:16.916424 25708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:16.916746 25708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:16.919807 25708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "787891ffeb084a6991155327c745b09f"
format_stamp: "Formatted at 2026-09-26 02:53:16 on dist-test-slave-ktng"
server_key: "0920a735b3a00e88456a293025682a3f"
server_key_iv: "4d95c54ff3fe05c964c6f0ddf8b05eed"
server_key_version: "encryptionkey@0"
I20260926 02:53:16.920605 25708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "787891ffeb084a6991155327c745b09f"
format_stamp: "Formatted at 2026-09-26 02:53:16 on dist-test-slave-ktng"
server_key: "0920a735b3a00e88456a293025682a3f"
server_key_iv: "4d95c54ff3fe05c964c6f0ddf8b05eed"
server_key_version: "encryptionkey@0"
I20260926 02:53:16.926362 25708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:53:16.930073 25722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:16.931455 25708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:53:16.931638 25708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "787891ffeb084a6991155327c745b09f"
format_stamp: "Formatted at 2026-09-26 02:53:16 on dist-test-slave-ktng"
server_key: "0920a735b3a00e88456a293025682a3f"
server_key_iv: "4d95c54ff3fe05c964c6f0ddf8b05eed"
server_key_version: "encryptionkey@0"
I20260926 02:53:16.931859 25708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:16.955607 25708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:16.956431 25708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:16.956722 25708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:16.974956 25708 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:43971
I20260926 02:53:16.975036 25773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:43971 every 8 connection(s)
I20260926 02:53:16.976292 25708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:53:16.980068 25774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:16.983657 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 25708
I20260926 02:53:16.983878 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:53:16.984531 20199 external_mini_cluster.cc:1428] Setting key 230a8d1f998a24a26f40031a0f420015
I20260926 02:53:16.992342 25774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f: Bootstrap starting.
I20260926 02:53:16.995921 25774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:16.997156 25774 log.cc:824] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:17.000371 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:16.986466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.19.185.254:43971)
0926 02:53:16.987509 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:16.987526 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:16.987578 (+ 52us) server_negotiation.cc:407] Connection header received
0926 02:53:16.990464 (+ 2886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:16.990503 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:16.991082 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:16.991557 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:16.992093 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:16.994074 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:16.995422 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:16.996519 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:16.996677 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:16.997762 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:16.997772 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:16.997795 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:53:16.998382 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:16.998496 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:53:16.998523 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:16.998634 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:16.998853 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 02:53:16.999141 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":297,"threads_started":1}
I20260926 02:53:17.000423 25774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f: No bootstrap required, opened a new log
I20260926 02:53:17.004087 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:53:17.008656 25774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787891ffeb084a6991155327c745b09f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43971 } }
I20260926 02:53:17.009029 25774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:17.009143 25774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787891ffeb084a6991155327c745b09f, State: Initialized, Role: FOLLOWER
I20260926 02:53:17.009791 25774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787891ffeb084a6991155327c745b09f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43971 } }
I20260926 02:53:17.010070 25774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:53:17.010212 25774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:53:17.010396 25774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:17.012632 25774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787891ffeb084a6991155327c745b09f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43971 } }
I20260926 02:53:17.013134 25774 leader_election.cc:302] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 787891ffeb084a6991155327c745b09f; no voters:
I20260926 02:53:17.013715 25774 leader_election.cc:288] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:53:17.013866 25779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:17.014602 25779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [term 1 LEADER]: Becoming Leader. State: Replica: 787891ffeb084a6991155327c745b09f, State: Running, Role: LEADER
I20260926 02:53:17.015224 25779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787891ffeb084a6991155327c745b09f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43971 } }
I20260926 02:53:17.016003 25774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:17.018261 25780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "787891ffeb084a6991155327c745b09f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787891ffeb084a6991155327c745b09f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43971 } } }
I20260926 02:53:17.018255 25781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 787891ffeb084a6991155327c745b09f. Latest consensus state: current_term: 1 leader_uuid: "787891ffeb084a6991155327c745b09f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787891ffeb084a6991155327c745b09f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43971 } } }
I20260926 02:53:17.018623 25780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:17.018740 25781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:17.020325 25785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:53:17.025398 25785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:53:17.035180 25785 catalog_manager.cc:1347] Generated new cluster ID: bea7cc685a4042459d9f82a88dfa9003
I20260926 02:53:17.035327 25785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:53:17.045605 25785 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:53:17.046880 25785 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:53:17.064056 25785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 787891ffeb084a6991155327c745b09f: Generated new TSK 0
I20260926 02:53:17.064913 25785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:53:17.537665 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:17.077173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.19.185.254:43971)
0926 02:53:17.077353 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:17.077359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:17.077716 (+ 357us) server_negotiation.cc:407] Connection header received
0926 02:53:17.077921 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:17.077937 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:17.078028 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:17.078196 (+ 168us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:53:17.078597 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:17.079802 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:17.081132 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:17.081822 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:17.081953 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:17.082380 (+ 427us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:53:17.537094 (+454714us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 02:53:17.537438 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46992: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 02:53:17.537791 25777 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46992: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 02:53:17.538007 25803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:17.077048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43971 (local address 127.0.0.1:46992)
0926 02:53:17.077617 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:17.077632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:17.077747 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:17.078211 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:17.078214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:17.078224 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:53:17.078449 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:17.078457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:17.079939 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:17.079943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:17.080986 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:17.080998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:17.081979 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:17.081983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:17.082096 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:17.082135 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:53:17.537406 (+455271us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 02:53:17.537658 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.185.254:43971: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":308,"threads_started":1}
I20260926 02:53:17.539141 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 25708
I20260926 02:53:17.540223 25769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:17.643250 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 25708
2026-09-26T02:53:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1151 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (408 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (423 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:53:18 dist-test-slave-ktng krb5kdc[25818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:53:18 dist-test-slave-ktng krb5kdc[25818](info): set up 2 sockets
Sep 26 02:53:18 dist-test-slave-ktng krb5kdc[25818](info): commencing operation
krb5kdc: starting...
W20260926 02:53:19.569090 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.005s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:53:19 dist-test-slave-ktng krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:53:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:53:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:53:19.782776 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:46069
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:46069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:20.105792 25831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:20.106087 25831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:20.106177 25831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:20.113345 25831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:20.113454 25831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:20.113502 25831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:20.113550 25831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:20.113586 25831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:53:20.123405 25831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:46069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:46069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:20.124688 25831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:20.125924 25831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:20.137825 25839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:20.137982 25836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:20.138020 25837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:20.138702 25831 server_base.cc:1034] running on GCE node
I20260926 02:53:20.139549 25831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:20.140866 25831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:20.142135 25831 hybrid_clock.cc:648] HybridClock initialized: now 1790391200142099 us; error 26 us; skew 500 ppm
I20260926 02:53:20.142496 25831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:20 dist-test-slave-ktng krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391200, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:20.145767 25831 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:53:20.147275 25831 webserver.cc:460] Webserver started at http://127.19.185.254:37253/ using document root <none> and password file <none>
I20260926 02:53:20.148010 25831 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:20.148159 25831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:20.148471 25831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:20.151444 25831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "601936d109cc40cfbd4f0d81f601d5bf"
format_stamp: "Formatted at 2026-09-26 02:53:20 on dist-test-slave-ktng"
server_key: "d8a658c05ff7a21a40bb49ef24c363c2"
server_key_iv: "8ad4fa2d3539cdd66ce5354ae9e3af0d"
server_key_version: "encryptionkey@0"
I20260926 02:53:20.152225 25831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "601936d109cc40cfbd4f0d81f601d5bf"
format_stamp: "Formatted at 2026-09-26 02:53:20 on dist-test-slave-ktng"
server_key: "d8a658c05ff7a21a40bb49ef24c363c2"
server_key_iv: "8ad4fa2d3539cdd66ce5354ae9e3af0d"
server_key_version: "encryptionkey@0"
I20260926 02:53:20.157298 25831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 02:53:20.160845 25846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:20.162163 25831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 02:53:20.162349 25831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "601936d109cc40cfbd4f0d81f601d5bf"
format_stamp: "Formatted at 2026-09-26 02:53:20 on dist-test-slave-ktng"
server_key: "d8a658c05ff7a21a40bb49ef24c363c2"
server_key_iv: "8ad4fa2d3539cdd66ce5354ae9e3af0d"
server_key_version: "encryptionkey@0"
I20260926 02:53:20.162552 25831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:20.176887 25831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:20.180801 25831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:20.181113 25831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:20.198800 25831 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:46069
I20260926 02:53:20.198829 25897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:46069 every 8 connection(s)
I20260926 02:53:20.200227 25831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:53:20.201690 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 25831
I20260926 02:53:20.201901 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:53:20.202740 20199 external_mini_cluster.cc:1428] Setting key f28c72ea75dd88306a9163c50ee949e8
I20260926 02:53:20.204710 25898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:20.215523 25898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf: Bootstrap starting.
Sep 26 02:53:20 dist-test-slave-ktng krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:20.218811 25898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:20.219956 25898 log.cc:824] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:20.222699 25898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf: No bootstrap required, opened a new log
I20260926 02:53:20.224628 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:20.204866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.19.185.254:46069)
0926 02:53:20.205586 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:20.205598 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:20.205655 (+ 57us) server_negotiation.cc:407] Connection header received
0926 02:53:20.207215 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:20.207236 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:20.207692 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:20.208078 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:20.209527 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:20.210822 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:20.212135 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:20.212914 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:20.213124 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:20.217445 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:20.217453 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:20.217466 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:20.217504 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:20.220758 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:20.221455 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:20.221469 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:20.221475 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:20.221637 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:20.222003 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:20.222009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:20.222012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:20.222499 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:20.222802 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:20.223254 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 02:53:20.223670 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":316,"threads_started":1}
I20260926 02:53:20.230401 25898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601936d109cc40cfbd4f0d81f601d5bf" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46069 } }
I20260926 02:53:20.230783 25898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:20.230885 25898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601936d109cc40cfbd4f0d81f601d5bf, State: Initialized, Role: FOLLOWER
I20260926 02:53:20.231513 25898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601936d109cc40cfbd4f0d81f601d5bf" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46069 } }
I20260926 02:53:20.231796 25898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:53:20.231950 25898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:53:20.232126 25898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:20.234202 25898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601936d109cc40cfbd4f0d81f601d5bf" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46069 } }
I20260926 02:53:20.234680 25898 leader_election.cc:302] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 601936d109cc40cfbd4f0d81f601d5bf; no voters:
I20260926 02:53:20.235286 25898 leader_election.cc:288] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:53:20.235435 25903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:20.236549 25903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [term 1 LEADER]: Becoming Leader. State: Replica: 601936d109cc40cfbd4f0d81f601d5bf, State: Running, Role: LEADER
I20260926 02:53:20.237063 25903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601936d109cc40cfbd4f0d81f601d5bf" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46069 } }
I20260926 02:53:20.237604 25898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:20.240027 25904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "601936d109cc40cfbd4f0d81f601d5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601936d109cc40cfbd4f0d81f601d5bf" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46069 } } }
I20260926 02:53:20.240051 25905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 601936d109cc40cfbd4f0d81f601d5bf. Latest consensus state: current_term: 1 leader_uuid: "601936d109cc40cfbd4f0d81f601d5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601936d109cc40cfbd4f0d81f601d5bf" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 46069 } } }
I20260926 02:53:20.240470 25904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:20.240470 25905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:20.244065 25910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:53:20.249156 25910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:53:20.257465 25910 catalog_manager.cc:1347] Generated new cluster ID: b1dbf47626ca4c4281afedb072286490
I20260926 02:53:20.257638 25910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:53:20.279453 25910 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:53:20.281361 25910 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:53:20.294123 25910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 601936d109cc40cfbd4f0d81f601d5bf: Generated new TSK 0
I20260926 02:53:20.295162 25910 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:53:20.347479 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46069
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:20.672044 25924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:20.672303 25924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:20.672469 25924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:20.679627 25924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:20.679759 25924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:20.679957 25924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:53:20.689831 25924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:20.691201 25924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:20.692524 25924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:20.706169 25932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:20.706141 25930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:20.706475 25929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:20.707108 25924 server_base.cc:1034] running on GCE node
I20260926 02:53:20.707716 25924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:20.708658 25924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:20.709882 25924 hybrid_clock.cc:648] HybridClock initialized: now 1790391200709764 us; error 107 us; skew 500 ppm
I20260926 02:53:20.710222 25924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:20 dist-test-slave-ktng krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391200, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:20.713403 25924 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:53:20.714757 25924 webserver.cc:460] Webserver started at http://127.19.185.193:35921/ using document root <none> and password file <none>
I20260926 02:53:20.715471 25924 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:20.715615 25924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:20.715929 25924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:20.718706 25924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "3ec7048012614fdcab21f5eb8a22c657"
format_stamp: "Formatted at 2026-09-26 02:53:20 on dist-test-slave-ktng"
server_key: "9cdc3b8158e733050f177715ca7212a5"
server_key_iv: "64bbe08979ec849367c881101443c3fb"
server_key_version: "encryptionkey@0"
I20260926 02:53:20.719522 25924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "3ec7048012614fdcab21f5eb8a22c657"
format_stamp: "Formatted at 2026-09-26 02:53:20 on dist-test-slave-ktng"
server_key: "9cdc3b8158e733050f177715ca7212a5"
server_key_iv: "64bbe08979ec849367c881101443c3fb"
server_key_version: "encryptionkey@0"
I20260926 02:53:20.724704 25924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:53:20.728158 25939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:20.729422 25924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:53:20.729588 25924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "3ec7048012614fdcab21f5eb8a22c657"
format_stamp: "Formatted at 2026-09-26 02:53:20 on dist-test-slave-ktng"
server_key: "9cdc3b8158e733050f177715ca7212a5"
server_key_iv: "64bbe08979ec849367c881101443c3fb"
server_key_version: "encryptionkey@0"
I20260926 02:53:20.729803 25924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:20.753202 25924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:20.757177 25924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:20.757470 25924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:20.758487 25924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:20.759835 25924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:20.759933 25924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:20.760075 25924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:20.760146 25924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:20.792832 25924 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:40493
I20260926 02:53:20.792882 26051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:40493 every 8 connection(s)
I20260926 02:53:20.794127 25924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:53:20.799259 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 25924
I20260926 02:53:20.799506 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:53:20.800140 20199 external_mini_cluster.cc:1428] Setting key b6f611ab72cd192f253d5d3fe058388f
Sep 26 02:53:20 dist-test-slave-ktng krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391200, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:20.815074 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:20.796642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:44201 (local address 127.19.185.254:46069)
0926 02:53:20.797095 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:20.797100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:20.798037 (+ 937us) server_negotiation.cc:407] Connection header received
0926 02:53:20.799998 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:20.800004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:20.800094 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:20.800213 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:20.801699 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:20.802398 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:20.803984 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:20.804380 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:20.804518 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:20.808113 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:20.808122 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:20.808131 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:20.808176 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:20.811757 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:20.812521 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:20.812529 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:20.812536 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:20.812703 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:20.813094 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:20.813102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:20.813107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:20.813514 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:20.813676 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:20.814346 (+ 670us) server_negotiation.cc:299] Negotiation successful
0926 02:53:20.814593 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":252,"threads_started":1}
I20260926 02:53:20.815716 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:20.797167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:46069 (local address 127.19.185.193:44201)
0926 02:53:20.797871 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:20.797909 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:20.799742 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:20.800450 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:20.800458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:20.800971 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:20.801531 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:20.801549 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:20.802556 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:20.802566 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:20.803844 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:20.803856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:20.804606 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:20.804613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:20.804856 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:20.805534 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:20.805556 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:20.807898 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:20.811894 (+ 3996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:20.811901 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:20.811922 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:20.812384 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:20.812809 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:20.812813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:20.812815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:20.812981 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:20.813648 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:20.813654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:20.813961 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 02:53:20.814186 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 02:53:20.814568 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":278,"threads_started":1}
I20260926 02:53:20.817345 26052 heartbeater.cc:347] Connected to a master server at 127.19.185.254:46069
I20260926 02:53:20.817711 26052 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:20.818692 26052 heartbeater.cc:511] Master 127.19.185.254:46069 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
I20260926 02:53:20.821089 25863 ts_manager.cc:194] Registered new tserver with Master: 3ec7048012614fdcab21f5eb8a22c657 (127.19.185.193:40493)
I20260926 02:53:20.822681 25863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:44201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:53:20.850715 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46069
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:21.176214 26058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:21.176491 26058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:21.176678 26058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:21.184173 26058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:21.184275 26058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:21.184477 26058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:53:21.194378 26058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:21.195715 26058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:21.197026 26058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:21.210412 26064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:21.210930 26063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:21.211762 26058 server_base.cc:1034] running on GCE node
W20260926 02:53:21.211912 26066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:21.212641 26058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:21.213919 26058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:21.215185 26058 hybrid_clock.cc:648] HybridClock initialized: now 1790391201215133 us; error 55 us; skew 500 ppm
I20260926 02:53:21.215540 26058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:21 dist-test-slave-ktng krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:21.218551 26058 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:53:21.220031 26058 webserver.cc:460] Webserver started at http://127.19.185.194:46007/ using document root <none> and password file <none>
I20260926 02:53:21.220700 26058 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:21.220795 26058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:21.221037 26058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:21.224097 26058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "17603ace394c4c0484cf0034e1aa7914"
format_stamp: "Formatted at 2026-09-26 02:53:21 on dist-test-slave-ktng"
server_key: "592ede2806427c3a3f2cb85ecb59f771"
server_key_iv: "e6e5606e6350e0aff52c32fc7b0f7b99"
server_key_version: "encryptionkey@0"
I20260926 02:53:21.224817 26058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "17603ace394c4c0484cf0034e1aa7914"
format_stamp: "Formatted at 2026-09-26 02:53:21 on dist-test-slave-ktng"
server_key: "592ede2806427c3a3f2cb85ecb59f771"
server_key_iv: "e6e5606e6350e0aff52c32fc7b0f7b99"
server_key_version: "encryptionkey@0"
I20260926 02:53:21.229786 26058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:53:21.233464 26073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:21.235137 26058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:53:21.235365 26058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "17603ace394c4c0484cf0034e1aa7914"
format_stamp: "Formatted at 2026-09-26 02:53:21 on dist-test-slave-ktng"
server_key: "592ede2806427c3a3f2cb85ecb59f771"
server_key_iv: "e6e5606e6350e0aff52c32fc7b0f7b99"
server_key_version: "encryptionkey@0"
I20260926 02:53:21.235546 26058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:21.264907 26058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:21.270017 26058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:21.270345 26058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:21.271448 26058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:21.272872 26058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:21.272961 26058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:21.273125 26058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:21.273186 26058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:21.308666 26058 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:45033
I20260926 02:53:21.308712 26185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:45033 every 8 connection(s)
I20260926 02:53:21.310105 26058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:53:21.310461 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 26058
I20260926 02:53:21.310621 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:53:21.311060 20199 external_mini_cluster.cc:1428] Setting key 7304f4022c68561015069274e173dd5b
Sep 26 02:53:21 dist-test-slave-ktng krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
I20260926 02:53:21.332787 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:21.313166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:46999 (local address 127.19.185.254:46069)
0926 02:53:21.313380 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:21.313385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:21.314814 (+ 1429us) server_negotiation.cc:407] Connection header received
0926 02:53:21.316676 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:21.316685 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:21.316783 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:21.316907 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:21.318403 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.319676 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.322279 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.322767 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.322875 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:21.326913 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:21.326917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:21.326921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:21.326953 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:21.329694 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:21.330426 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:21.330433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:21.330441 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:21.330566 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:21.330995 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:21.331000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:21.331003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:21.331268 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:21.331476 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:21.332203 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 02:53:21.332430 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:53:21.333911 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:21.313798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:46069 (local address 127.19.185.194:46999)
0926 02:53:21.314622 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:21.314674 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:21.316406 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:21.317113 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:21.317121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:21.317632 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:21.318222 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:21.318240 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.319825 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.319833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:21.322109 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:21.322126 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.322999 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.323008 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:21.323361 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:21.324068 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:21.324102 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:21.326689 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:21.329852 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:21.329861 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:21.329878 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:21.330294 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:21.330682 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:21.330687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:21.330691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:21.330879 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:21.331459 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:21.331467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:21.331813 (+ 346us) client_negotiation.cc:769] Sending connection context
0926 02:53:21.332089 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 02:53:21.332664 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":387,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
I20260926 02:53:21.335606 26186 heartbeater.cc:347] Connected to a master server at 127.19.185.254:46069
I20260926 02:53:21.335920 26186 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:21.336689 26186 heartbeater.cc:511] Master 127.19.185.254:46069 requested a full tablet report, sending...
I20260926 02:53:21.338213 25863 ts_manager.cc:194] Registered new tserver with Master: 17603ace394c4c0484cf0034e1aa7914 (127.19.185.194:45033)
I20260926 02:53:21.339192 25863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:46999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:53:21.359392 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46069
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:21.685149 26191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:21.685420 26191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:21.685520 26191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:21.693450 26191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:21.693581 26191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:21.693692 26191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:53:21.703679 26191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:46069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:21.704950 26191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:21.706212 26191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:21.719758 26196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:21.719774 26197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:21.719882 26199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:21.720811 26191 server_base.cc:1034] running on GCE node
I20260926 02:53:21.721421 26191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:21.722448 26191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:21.723676 26191 hybrid_clock.cc:648] HybridClock initialized: now 1790391201723624 us; error 46 us; skew 500 ppm
I20260926 02:53:21.724025 26191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:21 dist-test-slave-ktng krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:21.727442 26191 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:53:21.728963 26191 webserver.cc:460] Webserver started at http://127.19.185.195:43461/ using document root <none> and password file <none>
I20260926 02:53:21.729785 26191 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:21.729933 26191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:21.730290 26191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:21.733732 26191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "11efb46f2be34287945e03a7409df8ba"
format_stamp: "Formatted at 2026-09-26 02:53:21 on dist-test-slave-ktng"
server_key: "486fe58993ac0a12a594b024b118ac33"
server_key_iv: "7adcea1e24348adde42e4d5d72a7d70e"
server_key_version: "encryptionkey@0"
I20260926 02:53:21.734586 26191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "11efb46f2be34287945e03a7409df8ba"
format_stamp: "Formatted at 2026-09-26 02:53:21 on dist-test-slave-ktng"
server_key: "486fe58993ac0a12a594b024b118ac33"
server_key_iv: "7adcea1e24348adde42e4d5d72a7d70e"
server_key_version: "encryptionkey@0"
I20260926 02:53:21.740936 26191 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 02:53:21.744751 26206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:21.746194 26191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:53:21.746397 26191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "11efb46f2be34287945e03a7409df8ba"
format_stamp: "Formatted at 2026-09-26 02:53:21 on dist-test-slave-ktng"
server_key: "486fe58993ac0a12a594b024b118ac33"
server_key_iv: "7adcea1e24348adde42e4d5d72a7d70e"
server_key_version: "encryptionkey@0"
I20260926 02:53:21.746587 26191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:21.785203 26191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:21.789486 26191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:21.789829 26191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:21.790884 26191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:21.792389 26191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:21.792469 26191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:21.792618 26191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:21.792673 26191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:21.825917 26052 heartbeater.cc:503] Master 127.19.185.254:46069 was elected leader, sending a full tablet report...
I20260926 02:53:21.829634 26191 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:40759
I20260926 02:53:21.829675 26318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:40759 every 8 connection(s)
I20260926 02:53:21.830976 26191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:53:21.839221 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 26191
I20260926 02:53:21.839433 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:53:21.840077 20199 external_mini_cluster.cc:1428] Setting key 6245cfa3b98620388fbe9a0e9b328619
Sep 26 02:53:21 dist-test-slave-ktng krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:21.852808 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:21.833677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:53253 (local address 127.19.185.254:46069)
0926 02:53:21.834196 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:21.834203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:21.835397 (+ 1194us) server_negotiation.cc:407] Connection header received
0926 02:53:21.837175 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:21.837180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:21.837249 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:21.837413 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:21.838966 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.840382 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.842324 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.842737 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.842869 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:21.846737 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:21.846742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:21.846745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:21.846777 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:21.849470 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:21.850230 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:21.850234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:21.850236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:21.850363 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:21.850843 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:21.850846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:21.850849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:21.851195 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:21.851368 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:21.852139 (+ 771us) server_negotiation.cc:299] Negotiation successful
0926 02:53:21.852384 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":248,"threads_started":1}
I20260926 02:53:21.854063 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:21.834500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:46069 (local address 127.19.185.195:53253)
0926 02:53:21.835207 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:21.835253 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:21.836904 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:21.837579 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:21.837587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:21.838189 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:21.838787 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:21.838804 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.840570 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.840576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:21.842185 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:21.842197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.842961 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.842968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:21.843200 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:21.843950 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:21.843975 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:21.846535 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:21.849604 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:21.849613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:21.849627 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:21.850069 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:21.850473 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:21.850476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:21.850479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:21.850680 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:21.851366 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:21.851373 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:21.851705 (+ 332us) client_negotiation.cc:769] Sending connection context
0926 02:53:21.852051 (+ 346us) client_negotiation.cc:240] Negotiation successful
0926 02:53:21.852607 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":291,"threads_started":1}
I20260926 02:53:21.855760 26319 heartbeater.cc:347] Connected to a master server at 127.19.185.254:46069
I20260926 02:53:21.856139 26319 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:21.856868 26319 heartbeater.cc:511] Master 127.19.185.254:46069 requested a full tablet report, sending...
I20260926 02:53:21.858332 25862 ts_manager.cc:194] Registered new tserver with Master: 11efb46f2be34287945e03a7409df8ba (127.19.185.195:40759)
I20260926 02:53:21.859346 25862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:53253
I20260926 02:53:21.868040 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/test-admin.keytab.
Sep 26 02:53:21 dist-test-slave-ktng krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:21.893894 20199 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 02:53:21.899399 26327 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:21.896820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.19.185.193:40493)
0926 02:53:21.897493 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:21.897501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:21.897527 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:53:21.897590 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:21.897597 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:21.897800 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:21.898005 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:21.898927 (+ 922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47832: BlockingRecv error: recv got EOF from 127.0.0.1:47832 (error 108)
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":351,"threads_started":1}
W20260926 02:53:21.899504 20199 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/krb5cc' of type FILE
Sep 26 02:53:21 dist-test-slave-ktng krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:21.901072 20199 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 02:53:21 dist-test-slave-ktng krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:21.902618 20199 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 02:53:21 dist-test-slave-ktng krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:53:21.917835 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:21.904184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.19.185.193:40493)
0926 02:53:21.904391 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:21.904397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:21.904584 (+ 187us) server_negotiation.cc:407] Connection header received
0926 02:53:21.904732 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:21.904735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:21.904794 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:21.904877 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:21.905801 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.906728 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.907963 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.908498 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.908600 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:21.912340 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:21.912351 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:21.912366 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:21.912404 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:21.915153 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:21.915896 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:21.915903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:21.915905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:21.916024 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:21.916433 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:21.916437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:21.916439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:21.916789 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:21.916995 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:21.917275 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 02:53:21.917534 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 02:53:21.923437 26327 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:21.921879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.19.185.193:40493)
0926 02:53:21.922102 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:21.922107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:21.922298 (+ 191us) server_negotiation.cc:407] Connection header received
0926 02:53:21.922412 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:21.922416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:21.922468 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:21.922543 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:21.923155 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47844: BlockingRecv error: recv got EOF from 127.0.0.1:47844 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 02:53:21.923893 20199 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/krb5cc)
Sep 26 02:53:21 dist-test-slave-ktng krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:21.925366 20199 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 02:53:21 dist-test-slave-ktng krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:21.926654 20199 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 02:53:21 dist-test-slave-ktng krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:53:21.946318 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:21.927867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.19.185.193:40493)
0926 02:53:21.928051 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:21.928060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:21.928325 (+ 265us) server_negotiation.cc:407] Connection header received
0926 02:53:21.928432 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:21.928435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:21.928483 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:21.928558 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:21.929389 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.930051 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.931213 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.931934 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.932073 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:21.941070 (+ 8997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:21.941076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:21.941079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:21.941149 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:21.944019 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:21.944654 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:21.944658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:21.944660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:21.944782 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:21.945126 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:21.945129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:21.945131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:21.945388 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:21.945581 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:21.945811 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 02:53:21.946000 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 02:53:21.950601 26327 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:21.948979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.19.185.193:40493)
0926 02:53:21.949164 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:21.949170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:21.949373 (+ 203us) server_negotiation.cc:407] Connection header received
0926 02:53:21.949494 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:21.949497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:21.949561 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:21.949656 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:21.950384 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47864: BlockingRecv error: recv got EOF from 127.0.0.1:47864 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 02:53:21.950764 20199 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391151502216-20199-0/krb5kdc/krb5cc' of type FILE
Sep 26 02:53:21 dist-test-slave-ktng krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:21.952483 20199 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 02:53:21 dist-test-slave-ktng krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:21.953723 20199 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 02:53:21 dist-test-slave-ktng krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:53:21.966794 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:21.954992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.19.185.193:40493)
0926 02:53:21.955175 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:21.955183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:21.955627 (+ 444us) server_negotiation.cc:407] Connection header received
0926 02:53:21.955734 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:21.955737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:21.955795 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:21.955876 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:21.956813 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.957498 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.958654 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.959036 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.959131 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:21.962246 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:21.962250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:21.962253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:21.962279 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:21.964625 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:21.965137 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:21.965140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:21.965142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:21.965274 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:21.965578 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:21.965581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:21.965582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:21.965822 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:21.966104 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:21.966276 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 02:53:21.966420 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:53:21.968482 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 25924
I20260926 02:53:21.979269 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:21.969145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.19.185.193:40493)
0926 02:53:21.969325 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:21.969334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:21.969611 (+ 277us) server_negotiation.cc:407] Connection header received
0926 02:53:21.969726 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:21.969729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:21.969777 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:21.969849 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:21.970638 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.971264 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.972395 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:21.972794 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:21.972880 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:21.974693 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:21.974697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:21.974700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:21.974723 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:21.977062 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:21.977606 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:21.977610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:21.977612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:21.977737 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:21.978087 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:21.978090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:21.978092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:21.978316 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:21.978499 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:21.978763 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 02:53:21.978949 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 02:53:21.979895 26047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:22.064845 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 25924
I20260926 02:53:22.081373 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 26058
Sep 26 02:53:22 dist-test-slave-ktng krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:53:22.096274 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:22.082363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53790 (local address 127.19.185.194:45033)
0926 02:53:22.082859 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:22.082866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:22.082891 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:53:22.082948 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:22.082954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:22.083192 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:22.083372 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:22.084318 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:22.085309 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:22.086499 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:22.087009 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:22.087099 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:22.090800 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:22.090809 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:22.090823 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:22.090853 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:22.093744 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:22.094376 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:22.094383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:22.094385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:22.094494 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:22.094802 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:22.094804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:22.094806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:22.095113 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:22.095472 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:22.095677 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 02:53:22.095909 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":267,"threads_started":1}
I20260926 02:53:22.097482 26181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:22.180056 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 26058
I20260926 02:53:22.195554 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 26191
Sep 26 02:53:22 dist-test-slave-ktng krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:53:22.211475 26342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:22.196568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.19.185.195:40759)
0926 02:53:22.197220 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:22.197227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:22.197258 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:53:22.197320 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:22.197327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:22.197529 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:22.197732 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:22.198903 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:22.199984 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:22.201213 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:22.201766 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:22.201903 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:22.205839 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:22.205849 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:22.205864 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:22.205894 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:22.208619 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:22.209302 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:22.209309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:22.209311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:22.209430 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:22.209820 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:22.209826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:22.209829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:22.210171 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:22.210490 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:22.210765 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 02:53:22.211011 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":354,"threads_started":1}
I20260926 02:53:22.212826 26314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:22.296279 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 26191
I20260926 02:53:22.312233 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 25831
I20260926 02:53:22.313575 25893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:22.416857 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 25831
2026-09-26T02:53:22Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3947 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:53:22 dist-test-slave-ktng krb5kdc[26347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:53:22 dist-test-slave-ktng krb5kdc[26347](info): set up 2 sockets
Sep 26 02:53:22 dist-test-slave-ktng krb5kdc[26347](info): commencing operation
krb5kdc: starting...
W20260926 02:53:23.529104 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:53:23 dist-test-slave-ktng krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:53:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:53:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.254.keytab.
Entry for principal oryx/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.254.keytab.
I20260926 02:53:23.743250 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:45007
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.254.keytab
--principal=oryx/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:45007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:24.073227 26360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:24.073573 26360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:24.073745 26360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:24.081035 26360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:24.081188 26360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:24.081298 26360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:24.081401 26360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:24.081487 26360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:53:24.091187 26360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:45007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.254.keytab
--principal=oryx/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:45007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:24.092535 26360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:24.093832 26360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:24.105978 26365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:24.105952 26366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:24.106201 26368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:24.106923 26360 server_base.cc:1034] running on GCE node
I20260926 02:53:24.107753 26360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:24.109244 26360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:24.110507 26360 hybrid_clock.cc:648] HybridClock initialized: now 1790391204110425 us; error 64 us; skew 500 ppm
I20260926 02:53:24.110865 26360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:24 dist-test-slave-ktng krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391204, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:24.114184 26360 init.cc:377] Logged in from keytab as oryx/127.19.185.254@KRBTEST.COM (short username oryx)
I20260926 02:53:24.115907 26360 webserver.cc:460] Webserver started at http://127.19.185.254:33345/ using document root <none> and password file <none>
I20260926 02:53:24.116647 26360 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:24.116803 26360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:24.117125 26360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:24.120266 26360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "0987c8710d7a46eaaf9759293e32ffce"
format_stamp: "Formatted at 2026-09-26 02:53:24 on dist-test-slave-ktng"
server_key: "46182aa9576888fd3673afe060e74d89"
server_key_iv: "5a79454a6355ca3ad6fe271c7fcc2abe"
server_key_version: "encryptionkey@0"
I20260926 02:53:24.121053 26360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "0987c8710d7a46eaaf9759293e32ffce"
format_stamp: "Formatted at 2026-09-26 02:53:24 on dist-test-slave-ktng"
server_key: "46182aa9576888fd3673afe060e74d89"
server_key_iv: "5a79454a6355ca3ad6fe271c7fcc2abe"
server_key_version: "encryptionkey@0"
I20260926 02:53:24.126398 26360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:53:24.129879 26375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:24.131261 26360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:53:24.131479 26360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "0987c8710d7a46eaaf9759293e32ffce"
format_stamp: "Formatted at 2026-09-26 02:53:24 on dist-test-slave-ktng"
server_key: "46182aa9576888fd3673afe060e74d89"
server_key_iv: "5a79454a6355ca3ad6fe271c7fcc2abe"
server_key_version: "encryptionkey@0"
I20260926 02:53:24.131681 26360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:24.150309 26360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:24.154577 26360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:24.154889 26360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:24.173718 26360 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:45007
I20260926 02:53:24.173764 26426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:45007 every 8 connection(s)
I20260926 02:53:24.175077 26360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:53:24.178762 26427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:24.183828 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 26360
I20260926 02:53:24.184026 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:53:24.184834 20199 external_mini_cluster.cc:1428] Setting key 6c3200837d42a2d71c5985ca4acd67a3
I20260926 02:53:24.190032 26427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce: Bootstrap starting.
I20260926 02:53:24.193619 26427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:24.194854 26427 log.cc:824] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:24.198287 26427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce: No bootstrap required, opened a new log
Sep 26 02:53:24 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
I20260926 02:53:24.208724 26427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } }
I20260926 02:53:24.209156 26427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:24.209275 26427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0987c8710d7a46eaaf9759293e32ffce, State: Initialized, Role: FOLLOWER
I20260926 02:53:24.209556 26430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:24.187000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.19.185.254:45007)
0926 02:53:24.187783 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:24.187795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:24.187836 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:53:24.189485 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:24.189509 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:24.189888 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:24.190254 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:24.192175 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:24.193460 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:24.195219 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:24.196396 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:24.196524 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:24.200978 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:24.200988 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:24.201006 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:24.201066 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:24.204970 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:24.205627 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:24.205643 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:24.205651 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:24.205849 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:24.206257 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:24.206262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:24.206266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:24.207008 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:24.207330 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:24.207791 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 02:53:24.208195 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":412,"threads_started":1}
I20260926 02:53:24.209923 26427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } }
I20260926 02:53:24.210155 26427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:53:24.210307 26427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:53:24.210486 26427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:24.212803 26427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } }
I20260926 02:53:24.213274 26427 leader_election.cc:302] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0987c8710d7a46eaaf9759293e32ffce; no voters:
I20260926 02:53:24.213840 26427 leader_election.cc:288] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:53:24.213994 26432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:24.214779 26432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [term 1 LEADER]: Becoming Leader. State: Replica: 0987c8710d7a46eaaf9759293e32ffce, State: Running, Role: LEADER
I20260926 02:53:24.215385 26432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } }
I20260926 02:53:24.216063 26427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:24.218273 26434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0987c8710d7a46eaaf9759293e32ffce. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } }
I20260926 02:53:24.218600 26434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:24.219487 26433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } }
I20260926 02:53:24.219866 26433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:24.223407 26437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:53:24.229357 26437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:53:24.238077 26437 catalog_manager.cc:1347] Generated new cluster ID: c2353cf6506444608fba4eafd5e8bdb5
I20260926 02:53:24.238217 26437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:53:24.256492 26437 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:53:24.257850 26437 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:53:24.264511 26437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce: Generated new TSK 0
I20260926 02:53:24.265321 26437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.193.keytab.
Entry for principal oryx/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.193.keytab.
I20260926 02:53:24.327800 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:45007
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.193.keytab
--principal=oryx/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:24.667258 26453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:24.667651 26453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:24.667819 26453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:24.674942 26453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:24.675046 26453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:24.675251 26453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:53:24.684890 26453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.193.keytab
--principal=oryx/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:45007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:24.686116 26453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:24.687373 26453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:24.701103 26458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:24.701325 26459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:24.701103 26461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:24.702129 26453 server_base.cc:1034] running on GCE node
I20260926 02:53:24.702714 26453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:24.703702 26453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:24.704928 26453 hybrid_clock.cc:648] HybridClock initialized: now 1790391204704861 us; error 51 us; skew 500 ppm
I20260926 02:53:24.705276 26453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:24 dist-test-slave-ktng krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391204, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:24.708659 26453 init.cc:377] Logged in from keytab as oryx/127.19.185.193@KRBTEST.COM (short username oryx)
I20260926 02:53:24.710306 26453 webserver.cc:460] Webserver started at http://127.19.185.193:39099/ using document root <none> and password file <none>
I20260926 02:53:24.711045 26453 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:24.711205 26453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:24.711566 26453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:24.714633 26453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "ccf68e2b2b7249a6b858e3ea0a70588b"
format_stamp: "Formatted at 2026-09-26 02:53:24 on dist-test-slave-ktng"
server_key: "7e9f7375887975271f41cbd15d2314cf"
server_key_iv: "3d904e056a048153b802bf21d883e271"
server_key_version: "encryptionkey@0"
I20260926 02:53:24.715560 26453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "ccf68e2b2b7249a6b858e3ea0a70588b"
format_stamp: "Formatted at 2026-09-26 02:53:24 on dist-test-slave-ktng"
server_key: "7e9f7375887975271f41cbd15d2314cf"
server_key_iv: "3d904e056a048153b802bf21d883e271"
server_key_version: "encryptionkey@0"
I20260926 02:53:24.721189 26453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:53:24.724942 26468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:24.726480 26453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:53:24.726706 26453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "ccf68e2b2b7249a6b858e3ea0a70588b"
format_stamp: "Formatted at 2026-09-26 02:53:24 on dist-test-slave-ktng"
server_key: "7e9f7375887975271f41cbd15d2314cf"
server_key_iv: "3d904e056a048153b802bf21d883e271"
server_key_version: "encryptionkey@0"
I20260926 02:53:24.726934 26453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:24.751936 26453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:24.756831 26453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:24.757197 26453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:24.758285 26453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:24.759837 26453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:24.759950 26453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:24.760114 26453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:24.760193 26453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:24.796523 26453 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:43039
I20260926 02:53:24.796563 26580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:43039 every 8 connection(s)
I20260926 02:53:24.797994 26453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:53:24.801519 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 26453
I20260926 02:53:24.801731 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:53:24.802433 20199 external_mini_cluster.cc:1428] Setting key 54b5595fa2535f0d356be1fb77093ee5
Sep 26 02:53:24 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391204, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
I20260926 02:53:24.819650 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:24.800648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:50933 (local address 127.19.185.254:45007)
0926 02:53:24.801102 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:24.801106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:24.802110 (+ 1004us) server_negotiation.cc:407] Connection header received
0926 02:53:24.804546 (+ 2436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:24.804551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:24.804627 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:24.804744 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:24.806196 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:24.807342 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:24.808897 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:24.809643 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:24.809786 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:24.813384 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:24.813389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:24.813392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:24.813421 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:24.816305 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:24.817145 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:24.817150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:24.817154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:24.817323 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:24.817810 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:24.817816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:24.817819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:24.818115 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:24.818278 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:24.819044 (+ 766us) server_negotiation.cc:299] Negotiation successful
0926 02:53:24.819262 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":245,"threads_started":1}
I20260926 02:53:24.820677 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:24.801099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.193:50933)
0926 02:53:24.801937 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:24.801982 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:24.804235 (+ 2253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:24.804932 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:24.804943 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:24.805477 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:24.806028 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:24.806045 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:24.807519 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:24.807523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:24.808747 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:24.808759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:24.809874 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:24.809879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:24.810076 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:24.810806 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:24.810829 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:24.813174 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:24.816462 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:24.816471 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:24.816490 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:24.817021 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:24.817464 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:24.817469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:24.817473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:24.817699 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:24.818245 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:24.818255 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:24.818616 (+ 361us) client_negotiation.cc:769] Sending connection context
0926 02:53:24.818940 (+ 324us) client_negotiation.cc:240] Negotiation successful
0926 02:53:24.819545 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":379,"threads_started":1}
I20260926 02:53:24.822345 26581 heartbeater.cc:347] Connected to a master server at 127.19.185.254:45007
I20260926 02:53:24.822674 26581 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.19.185.194@KRBTEST.COM; defaulting to no policy
I20260926 02:53:24.823494 26581 heartbeater.cc:511] Master 127.19.185.254:45007 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.185.194@KRBTEST.COM" created.
I20260926 02:53:24.825796 26392 ts_manager.cc:194] Registered new tserver with Master: ccf68e2b2b7249a6b858e3ea0a70588b (127.19.185.193:43039)
I20260926 02:53:24.827273 26392 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:50933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.194.keytab.
Entry for principal oryx/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.194.keytab.
I20260926 02:53:24.850992 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:45007
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.194.keytab
--principal=oryx/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:25.171859 26587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:25.172119 26587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:25.172292 26587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:25.179649 26587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:25.179785 26587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:25.179962 26587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:53:25.189452 26587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.194.keytab
--principal=oryx/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:45007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:25.190666 26587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:25.192034 26587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:25.204591 26592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:25.204588 26595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:25.204543 26593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:25.205490 26587 server_base.cc:1034] running on GCE node
I20260926 02:53:25.206064 26587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:25.206902 26587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:25.208089 26587 hybrid_clock.cc:648] HybridClock initialized: now 1790391205208047 us; error 30 us; skew 500 ppm
I20260926 02:53:25.208395 26587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:25 dist-test-slave-ktng krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391205, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:25.211544 26587 init.cc:377] Logged in from keytab as oryx/127.19.185.194@KRBTEST.COM (short username oryx)
I20260926 02:53:25.212932 26587 webserver.cc:460] Webserver started at http://127.19.185.194:36443/ using document root <none> and password file <none>
I20260926 02:53:25.213580 26587 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:25.213703 26587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:25.214001 26587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:25.216852 26587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "9cf04e24574d4f32ac96afb67043ca5a"
format_stamp: "Formatted at 2026-09-26 02:53:25 on dist-test-slave-ktng"
server_key: "a55b4e19e5cc83c5604abb816d9089fe"
server_key_iv: "ab0f757194987440636998e8d6abeffb"
server_key_version: "encryptionkey@0"
I20260926 02:53:25.217644 26587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "9cf04e24574d4f32ac96afb67043ca5a"
format_stamp: "Formatted at 2026-09-26 02:53:25 on dist-test-slave-ktng"
server_key: "a55b4e19e5cc83c5604abb816d9089fe"
server_key_iv: "ab0f757194987440636998e8d6abeffb"
server_key_version: "encryptionkey@0"
I20260926 02:53:25.222462 26587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 02:53:25.225966 26602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:25.227419 26587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:53:25.227602 26587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "9cf04e24574d4f32ac96afb67043ca5a"
format_stamp: "Formatted at 2026-09-26 02:53:25 on dist-test-slave-ktng"
server_key: "a55b4e19e5cc83c5604abb816d9089fe"
server_key_iv: "ab0f757194987440636998e8d6abeffb"
server_key_version: "encryptionkey@0"
I20260926 02:53:25.227807 26587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:25.243125 26587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:25.247555 26587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:25.247890 26587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:25.248843 26587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:25.250286 26587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:25.250388 26587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:25.250531 26587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:25.250607 26587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:25.283278 26587 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:34167
I20260926 02:53:25.283489 26714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:34167 every 8 connection(s)
I20260926 02:53:25.284631 26587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:53:25.290561 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 26587
I20260926 02:53:25.290740 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:53:25.291179 20199 external_mini_cluster.cc:1428] Setting key 8f716433cfe6a9ef4a6091ab47baa3d4
Sep 26 02:53:25 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391205, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
I20260926 02:53:25.306162 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:25.287121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:45347 (local address 127.19.185.254:45007)
0926 02:53:25.287376 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:25.287380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:25.288541 (+ 1161us) server_negotiation.cc:407] Connection header received
0926 02:53:25.290303 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:25.290308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:25.290401 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:25.290509 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:25.292223 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:25.293024 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:25.295547 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:25.295979 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:25.296073 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:25.300161 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:25.300169 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:25.300175 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:25.300226 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:25.302971 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:25.303931 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:25.303934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:25.303936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:25.304056 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:25.304537 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:25.304539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:25.304541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:25.304800 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:25.304904 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:25.305765 (+ 861us) server_negotiation.cc:299] Negotiation successful
0926 02:53:25.305927 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:53:25.307770 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:25.287639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.194:45347)
0926 02:53:25.288378 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:25.288423 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:25.290014 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:25.290773 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:25.290781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:25.291387 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:25.292054 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:25.292074 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:25.293197 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:25.293202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:25.295406 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:25.295424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:25.296209 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:25.296220 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:25.296538 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:25.297260 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:25.297292 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:25.299908 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:25.303121 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:25.303132 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:25.303152 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:25.303816 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:25.304160 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:25.304168 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:25.304175 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:25.304425 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:25.304920 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:25.304930 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:25.305371 (+ 441us) client_negotiation.cc:769] Sending connection context
0926 02:53:25.305676 (+ 305us) client_negotiation.cc:240] Negotiation successful
0926 02:53:25.306251 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":263,"threads_started":1}
I20260926 02:53:25.309336 26715 heartbeater.cc:347] Connected to a master server at 127.19.185.254:45007
I20260926 02:53:25.309679 26715 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:25.310416 26715 heartbeater.cc:511] Master 127.19.185.254:45007 requested a full tablet report, sending...
I20260926 02:53:25.311920 26392 ts_manager.cc:194] Registered new tserver with Master: 9cf04e24574d4f32ac96afb67043ca5a (127.19.185.194:34167)
I20260926 02:53:25.312708 26392 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:45347
WARNING: no policy specified for oryx/127.19.185.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.195.keytab.
Entry for principal oryx/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.195.keytab.
I20260926 02:53:25.343770 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:45007
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.195.keytab
--principal=oryx/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:25.667882 26720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:25.668170 26720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:25.668263 26720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:25.675343 26720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:25.675453 26720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:25.675571 26720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:53:25.685453 26720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.195.keytab
--principal=oryx/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:45007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:25.686677 26720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:25.688054 26720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:25.701421 26728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:25.701447 26726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:25.701915 26725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:25.702399 26720 server_base.cc:1034] running on GCE node
I20260926 02:53:25.703094 26720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:25.704110 26720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:25.705303 26720 hybrid_clock.cc:648] HybridClock initialized: now 1790391205705265 us; error 35 us; skew 500 ppm
I20260926 02:53:25.705610 26720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:25 dist-test-slave-ktng krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391205, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:25.708730 26720 init.cc:377] Logged in from keytab as oryx/127.19.185.195@KRBTEST.COM (short username oryx)
I20260926 02:53:25.710165 26720 webserver.cc:460] Webserver started at http://127.19.185.195:42783/ using document root <none> and password file <none>
I20260926 02:53:25.710837 26720 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:25.710990 26720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:25.711347 26720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:25.714128 26720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "855525e436144877b3db94a22a28141f"
format_stamp: "Formatted at 2026-09-26 02:53:25 on dist-test-slave-ktng"
server_key: "bf77a8478ddffedb0f0b360e180fcfc7"
server_key_iv: "739c0a15005465bfdcce000ffac9183b"
server_key_version: "encryptionkey@0"
I20260926 02:53:25.714928 26720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "855525e436144877b3db94a22a28141f"
format_stamp: "Formatted at 2026-09-26 02:53:25 on dist-test-slave-ktng"
server_key: "bf77a8478ddffedb0f0b360e180fcfc7"
server_key_iv: "739c0a15005465bfdcce000ffac9183b"
server_key_version: "encryptionkey@0"
I20260926 02:53:25.720355 26720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:53:25.723748 26735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:25.725255 26720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 02:53:25.725419 26720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "855525e436144877b3db94a22a28141f"
format_stamp: "Formatted at 2026-09-26 02:53:25 on dist-test-slave-ktng"
server_key: "bf77a8478ddffedb0f0b360e180fcfc7"
server_key_iv: "739c0a15005465bfdcce000ffac9183b"
server_key_version: "encryptionkey@0"
I20260926 02:53:25.725618 26720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:25.745159 26720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:25.749284 26720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:25.749581 26720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:25.750540 26720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:25.751931 26720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:25.752034 26720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:25.752190 26720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:25.752262 26720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:25.786157 26720 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:40521
I20260926 02:53:25.786247 26847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:40521 every 8 connection(s)
I20260926 02:53:25.787448 26720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:53:25.794201 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 26720
I20260926 02:53:25.794384 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:53:25.794988 20199 external_mini_cluster.cc:1428] Setting key 955d826da7f5d4f125211c243225e5ed
Sep 26 02:53:25 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391205, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
I20260926 02:53:25.807505 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:25.790026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:50183 (local address 127.19.185.254:45007)
0926 02:53:25.790216 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:25.790220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:25.791345 (+ 1125us) server_negotiation.cc:407] Connection header received
0926 02:53:25.793125 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:25.793132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:25.793245 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:25.793359 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:25.795369 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:25.796507 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:25.798569 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:25.799200 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:25.799342 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:25.802641 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:25.802646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:25.802649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:25.802678 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:25.805032 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:25.805645 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:25.805649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:25.805651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:25.805764 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:25.806113 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:25.806115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:25.806117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:25.806363 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:25.806466 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:25.807051 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 02:53:25.807211 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:53:25.808504 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:25.790472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.195:50183)
0926 02:53:25.791168 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:25.791210 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:25.792851 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:25.793591 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:25.793599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:25.794325 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:25.795135 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:25.795163 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:25.796698 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:25.796703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:25.798425 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:25.798437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:25.799446 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:25.799451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:25.799641 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:25.800384 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:25.800412 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:25.802461 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:25.805149 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:25.805156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:25.805167 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:25.805549 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:25.805852 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:25.805855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:25.805857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:25.806023 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:25.806460 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:25.806465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:25.806753 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 02:53:25.806970 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 02:53:25.807385 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":284,"threads_started":1}
I20260926 02:53:25.810047 26848 heartbeater.cc:347] Connected to a master server at 127.19.185.254:45007
I20260926 02:53:25.810364 26848 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:25.811105 26848 heartbeater.cc:511] Master 127.19.185.254:45007 requested a full tablet report, sending...
I20260926 02:53:25.812500 26392 ts_manager.cc:194] Registered new tserver with Master: 855525e436144877b3db94a22a28141f (127.19.185.195:40521)
I20260926 02:53:25.813282 26392 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:50183
I20260926 02:53:25.822786 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:53:25.830960 26581 heartbeater.cc:503] Master 127.19.185.254:45007 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.19.185.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.185.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.185.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.253.keytab.
Entry for principal oryx/127.19.185.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.253.keytab.
I20260926 02:53:25.875916 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.253:34313
--webserver_interface=127.19.185.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.253.keytab
--principal=oryx/127.19.185.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:45007,127.19.185.253:34313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:26.203928 26853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:26.204226 26853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:26.204382 26853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:26.211681 26853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:26.211789 26853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:26.211896 26853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:26.212003 26853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:26.212083 26853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:53:26.222103 26853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:45007,127.19.185.253:34313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.253.keytab
--principal=oryx/127.19.185.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.253:34313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.185.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
W20260926 02:53:26.222492 26853 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.19.185.254:45007,127.19.185.253:34313'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 02:53:26.223592 26853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:26.224987 26853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:26.237227 26859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:26.237195 26861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:26.237600 26858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:26.238391 26853 server_base.cc:1034] running on GCE node
I20260926 02:53:26.239168 26853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:26.240594 26853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:26.241846 26853 hybrid_clock.cc:648] HybridClock initialized: now 1790391206241746 us; error 86 us; skew 500 ppm
I20260926 02:53:26.242183 26853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:26 dist-test-slave-ktng krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391206, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:26.245720 26853 init.cc:377] Logged in from keytab as oryx/127.19.185.253@KRBTEST.COM (short username oryx)
I20260926 02:53:26.247438 26853 webserver.cc:460] Webserver started at http://127.19.185.253:36945/ using document root <none> and password file <none>
I20260926 02:53:26.248234 26853 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:26.248391 26853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:26.248744 26853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:26.252132 26853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/data/instance:
uuid: "3bcc74bcc364436799b69b73517ab95c"
format_stamp: "Formatted at 2026-09-26 02:53:26 on dist-test-slave-ktng"
server_key: "69e6a2260a2e0c2486c0dd6353e97747"
server_key_iv: "731d8539f84db73cd91584c6a530c77e"
server_key_version: "encryptionkey@0"
I20260926 02:53:26.252974 26853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/wal/instance:
uuid: "3bcc74bcc364436799b69b73517ab95c"
format_stamp: "Formatted at 2026-09-26 02:53:26 on dist-test-slave-ktng"
server_key: "69e6a2260a2e0c2486c0dd6353e97747"
server_key_iv: "731d8539f84db73cd91584c6a530c77e"
server_key_version: "encryptionkey@0"
I20260926 02:53:26.258452 26853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:53:26.261859 26868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:26.263142 26853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:53:26.263375 26853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/wal
uuid: "3bcc74bcc364436799b69b73517ab95c"
format_stamp: "Formatted at 2026-09-26 02:53:26 on dist-test-slave-ktng"
server_key: "69e6a2260a2e0c2486c0dd6353e97747"
server_key_iv: "731d8539f84db73cd91584c6a530c77e"
server_key_version: "encryptionkey@0"
I20260926 02:53:26.263584 26853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:26.279956 26853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:26.283833 26853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:26.284099 26853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:26.302279 26853 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.253:34313
I20260926 02:53:26.302299 26919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.253:34313 every 8 connection(s)
I20260926 02:53:26.303646 26853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/data/info.pb
I20260926 02:53:26.305912 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 26853
I20260926 02:53:26.307901 26920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:26.315171 26715 heartbeater.cc:503] Master 127.19.185.254:45007 was elected leader, sending a full tablet report...
I20260926 02:53:26.316735 26920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:53:26 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.185.253@KRBTEST.COM
Sep 26 02:53:26 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391206, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.253@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
I20260926 02:53:26.332198 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.308196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.19.185.253:34313)
0926 02:53:26.308963 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:26.308974 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:26.309015 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:53:26.310663 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:26.310684 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:26.311133 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:26.311510 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:26.313246 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.314587 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.316395 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.317408 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.317551 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.322628 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:26.322636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:26.322647 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:26.322683 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:26.326850 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.327645 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.327657 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.327666 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.327892 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.328318 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.328323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.328327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.329146 (+ 819us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:26.329433 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:26.330126 (+ 693us) server_negotiation.cc:299] Negotiation successful
0926 02:53:26.330602 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":370,"threads_started":1}
I20260926 02:53:26.336942 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.319730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46628 (local address 127.19.185.254:45007)
0926 02:53:26.320277 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:26.320284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:26.320798 (+ 514us) server_negotiation.cc:407] Connection header received
0926 02:53:26.321014 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:26.321021 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:26.321175 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:26.321296 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:26.322353 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.323154 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.325641 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.326038 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.326181 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.330496 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:26.330503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:26.330507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:26.330553 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:26.333866 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.334636 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.334644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.334651 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.334793 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.335220 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.335226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.335230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.335632 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:26.335801 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:26.336286 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 02:53:26.336534 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":265,"threads_started":1}
I20260926 02:53:26.336997 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.319768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:46628)
0926 02:53:26.320642 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:26.320688 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:26.320877 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:26.321349 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:26.321355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:26.321960 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:26.322204 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.322212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.323354 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.323359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.325486 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.325503 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.326294 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.326301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.326503 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.327236 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:26.327252 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:26.330226 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:26.334024 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:26.334035 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:26.334042 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:26.334503 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:26.334921 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:26.334928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:26.334931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:26.335082 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:26.335781 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:26.335789 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:26.335994 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 02:53:26.336190 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 02:53:26.336547 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":480,"threads_started":1}
I20260926 02:53:26.339080 26920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:53:26 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391206, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.253@KRBTEST.COM for oryx/127.19.185.253@KRBTEST.COM
I20260926 02:53:26.352108 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.339957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:34313 (local address 127.0.0.1:44280)
0926 02:53:26.340237 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:26.340251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:26.340342 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:26.340589 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:26.340592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:26.340902 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:26.341098 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.341106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.342125 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.342128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.342988 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.342999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.343751 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.343755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.343882 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.344467 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:26.344483 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:26.346535 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:26.349369 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:26.349372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:26.349375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:26.349761 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:26.350203 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:26.350209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:26.350213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:26.350361 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:26.350932 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:26.350937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:26.351176 (+ 239us) client_negotiation.cc:769] Sending connection context
0926 02:53:26.351366 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 02:53:26.351680 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260926 02:53:26.352219 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.339957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44280 (local address 127.19.185.253:34313)
0926 02:53:26.340155 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:26.340160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:26.340320 (+ 160us) server_negotiation.cc:407] Connection header received
0926 02:53:26.340416 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:26.340419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:26.340472 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:26.340546 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:26.341221 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.341909 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.343148 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.343605 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.343701 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.346729 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:26.346733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:26.346737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:26.346764 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:26.349237 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.349903 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.349907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.349910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.350059 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.350477 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.350493 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.350496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.350808 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:26.350969 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:26.351445 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 02:53:26.351833 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":100}
I20260926 02:53:26.357168 26920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: Bootstrap starting.
I20260926 02:53:26.360556 26920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:26.361629 26920 log.cc:824] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:26.364535 26920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: No bootstrap required, opened a new log
I20260926 02:53:26.372327 26920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } }
I20260926 02:53:26.372666 26920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:26.372778 26920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bcc74bcc364436799b69b73517ab95c, State: Initialized, Role: FOLLOWER
I20260926 02:53:26.373433 26920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } }
I20260926 02:53:26.374303 26927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } } }
I20260926 02:53:26.374636 26927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:53:26.375453 26920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: configured and running, proceeding with master startup.
W20260926 02:53:26.383828 26938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:53:26.383967 26938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:53:26.387174 26853 master_runner.cc:428] Detected that this master 3bcc74bcc364436799b69b73517ab95c is joining an existing cluster
I20260926 02:53:26.387275 26853 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.185.253:34313
WARNING: no policy specified for oryx/127.19.185.252@KRBTEST.COM; defaulting to no policy
I20260926 02:53:26.413646 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.394755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.19.185.253:34313)
0926 02:53:26.394988 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:26.394995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:26.397022 (+ 2027us) server_negotiation.cc:407] Connection header received
0926 02:53:26.397269 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:26.397276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:26.397348 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:26.397425 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:26.398964 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.400230 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.402360 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.403228 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.403363 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.406093 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:26.406102 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:26.406109 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:26.406149 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:26.410200 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.411091 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.411099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.411105 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.411280 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.411788 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.411795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.411799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.412269 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:26.412440 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:26.412936 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 02:53:26.413285 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":216,"server-negotiator.queue_time_us":61}
I20260926 02:53:26.413681 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.394741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:34313 (local address 127.0.0.1:44296)
0926 02:53:26.396922 (+ 2181us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:26.396942 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:26.397128 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:26.397630 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:26.397636 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:26.398402 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:26.398815 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.398830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.400437 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.400445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.402203 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.402223 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.403666 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.403673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.403906 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.404679 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:26.404710 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:26.405912 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:26.410353 (+ 4441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:26.410359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:26.410363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:26.410959 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:26.411465 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:26.411469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:26.411474 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:26.411658 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:26.412504 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:26.412508 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:26.412671 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 02:53:26.412850 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 02:53:26.413161 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2048,"thread_start_us":1915,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.185.252@KRBTEST.COM" created.
I20260926 02:53:26.423888 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.394706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46632 (local address 127.19.185.254:45007)
0926 02:53:26.394904 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:26.394912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:26.402877 (+ 7965us) server_negotiation.cc:407] Connection header received
0926 02:53:26.403178 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:26.403185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:26.403332 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:26.403415 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:26.405095 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.406290 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.412343 (+ 6053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.413093 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.413229 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.416448 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:26.416456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:26.416461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:26.416518 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:26.420670 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.421537 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.421546 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.421552 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.421705 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.422183 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.422191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.422196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.422585 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:26.422746 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:26.423252 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 02:53:26.423504 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:53:26.423959 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.395348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:46632)
0926 02:53:26.402747 (+ 7399us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:26.402769 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:26.402987 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:26.403744 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:26.403751 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:26.404509 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:26.404885 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.404898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.406526 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.406535 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.412158 (+ 5623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.412174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.413682 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.413690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.414505 (+ 815us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.415221 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:26.415247 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:26.416233 (+ 986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:26.420842 (+ 4609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:26.420850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:26.420857 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:26.421380 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:26.421861 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:26.421868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:26.421874 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:26.422043 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:26.422826 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:26.422831 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:26.422993 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 02:53:26.423146 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 02:53:26.423504 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7235,"mutex_wait_us":25,"thread_start_us":6748,"threads_started":1}
Sep 26 02:53:26 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391204, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.254@KRBTEST.COM for oryx/127.19.185.253@KRBTEST.COM
I20260926 02:53:26.445960 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.430708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.19.185.253:34313)
0926 02:53:26.430906 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:26.430913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:26.431566 (+ 653us) server_negotiation.cc:407] Connection header received
0926 02:53:26.432106 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:26.432113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:26.432303 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:26.432421 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:26.434018 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.434857 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.436511 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.436971 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.437114 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.440367 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:26.440372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:26.440375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:26.440404 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:26.443088 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.443854 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.443859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.443863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.444004 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.444419 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.444423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.444425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.444698 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:26.444855 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:26.445344 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 02:53:26.445553 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:53:26.446012 26948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.430705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:34313 (local address 127.0.0.1:44298)
0926 02:53:26.431443 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:26.431487 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:26.431686 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:26.432969 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:26.433004 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:26.433659 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:26.433868 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.433877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.434982 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.434989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.436377 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.436389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.437135 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.437141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.437349 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.437998 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:26.438016 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:26.440177 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:26.443269 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:26.443281 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:26.443289 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:26.443728 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:26.444123 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:26.444129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:26.444132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:26.444292 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:26.444833 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:26.444841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:26.445026 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 02:53:26.445220 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 02:53:26.445553 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":321,"threads_started":1}
I20260926 02:53:26.448098 26392 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.185.253:34313: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3bcc74bcc364436799b69b73517ab95c"
member_type: NON_VOTER
last_known_addr {
host: "127.19.185.253"
port: 34313
}
attrs {
promote: true
}
}
dest_uuid: "0987c8710d7a46eaaf9759293e32ffce"
cas_config_opid_index: -1
I20260926 02:53:26.449034 26392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: NON_VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: true } }
I20260926 02:53:26.452032 26951 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3bcc74bcc364436799b69b73517ab95c (127.19.185.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: NON_VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: true } } }
I20260926 02:53:26.453398 26895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:26.455610 26951 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.185.253:34313
I20260926 02:53:26.456311 26853 master.cc:537] Master@127.19.185.253:34313 shutting down...
I20260926 02:53:26.456401 26952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: NON_VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: true } } }
I20260926 02:53:26.457125 26952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.185.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.252.keytab.
Entry for principal oryx/127.19.185.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.252.keytab.
I20260926 02:53:26.459738 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.252:44145
--webserver_interface=127.19.185.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.252.keytab
--principal=oryx/127.19.185.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:45007,127.19.185.253:34313,127.19.185.252:44145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:53:26.461411 26895 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 FOLLOWER]: Refusing update from remote peer 0987c8710d7a46eaaf9759293e32ffce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 02:53:26.462702 26951 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: NON_VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 02:53:26.466063 26378 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 02:53:26.471976 26853 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:26.472443 26853 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:26.472663 26853 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: stopping tablet replica
W20260926 02:53:26.474102 26378 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce -> Peer 3bcc74bcc364436799b69b73517ab95c (127.19.185.253:34313): Couldn't send request to peer 3bcc74bcc364436799b69b73517ab95c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 02:53:26.492179 26853 master.cc:552] Master@127.19.185.253:34313 shutdown complete.
I20260926 02:53:26.492280 26853 master_runner.cc:305] Clearing existing system tablet
I20260926 02:53:26.492774 26853 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:26.492874 26853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:26.497210 26853 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 02:53:26.499076 26957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:26.499509 26853 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:26.499811 26853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260926 02:53:26.499907 26853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/wal
uuid: "3bcc74bcc364436799b69b73517ab95c"
format_stamp: "Formatted at 2026-09-26 02:53:26 on dist-test-slave-ktng"
server_key: "69e6a2260a2e0c2486c0dd6353e97747"
server_key_iv: "731d8539f84db73cd91584c6a530c77e"
server_key_version: "encryptionkey@0"
I20260926 02:53:26.505259 26853 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:26.512928 26853 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 02:53:26.513135 26853 log.cc:1196] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 02:53:26.513635 26853 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: Deleting consensus metadata
I20260926 02:53:26.513917 26853 master_runner.cc:315] Copying system tablet from 127.19.185.254:45007
I20260926 02:53:26.516927 26853 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: tablet copy: Beginning tablet copy session from remote peer at address 127.19.185.254:45007
I20260926 02:53:26.534042 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.517858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:46640)
0926 02:53:26.518582 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:26.518609 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:26.518795 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:26.519269 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:26.519274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:26.519891 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:26.520244 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.520265 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.521970 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.521977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.523725 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.523742 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.524758 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.524766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.524968 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.525742 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:26.525767 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:26.526912 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:26.530852 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:26.530859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:26.530864 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:26.531464 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:26.531957 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:26.531963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:26.531969 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:26.532147 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:26.532851 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:26.532956 (+ 105us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:26.533133 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 02:53:26.533301 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 02:53:26.533600 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":404,"threads_started":1}
I20260926 02:53:26.534135 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.517897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46640 (local address 127.19.185.254:45007)
0926 02:53:26.518116 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:26.518130 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:26.518725 (+ 595us) server_negotiation.cc:407] Connection header received
0926 02:53:26.518978 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:26.518985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:26.519088 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:26.519239 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:26.520449 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.521783 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.523901 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.524576 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.524716 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.527143 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:26.527150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:26.527154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:26.527201 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:26.530684 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.531597 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.531602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.531605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.531757 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.532307 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.532313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.532317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.532710 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:26.532869 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:26.533465 (+ 596us) server_negotiation.cc:299] Negotiation successful
0926 02:53:26.533708 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 02:53:26.535420 26412 tablet_copy_service.cc:140] P 0987c8710d7a46eaaf9759293e32ffce: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3bcc74bcc364436799b69b73517ab95c ({username='oryx', principal='oryx/127.19.185.253@KRBTEST.COM'} at 127.0.0.1:46640)
I20260926 02:53:26.535773 26412 tablet_copy_service.cc:161] P 0987c8710d7a46eaaf9759293e32ffce: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3bcc74bcc364436799b69b73517ab95c at {username='oryx', principal='oryx/127.19.185.253@KRBTEST.COM'} at 127.0.0.1:46640: session id = 3bcc74bcc364436799b69b73517ab95c-00000000000000000000000000000000
I20260926 02:53:26.541000 26412 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 02:53:26.543501 26853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 02:53:26.551525 26853 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: tablet copy: Starting download of 0 data blocks...
I20260926 02:53:26.551865 26853 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: tablet copy: Starting download of 1 WAL segments...
I20260926 02:53:26.554675 26853 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 02:53:26.559420 26412 tablet_copy_service.cc:342] P 0987c8710d7a46eaaf9759293e32ffce: Request end of tablet copy session 3bcc74bcc364436799b69b73517ab95c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.185.253@KRBTEST.COM'} at 127.0.0.1:46640
I20260926 02:53:26.559651 26412 tablet_copy_service.cc:433] P 0987c8710d7a46eaaf9759293e32ffce: ending tablet copy session 3bcc74bcc364436799b69b73517ab95c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3bcc74bcc364436799b69b73517ab95c
I20260926 02:53:26.571206 26853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:26.574123 26964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:26.574874 26965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:26.579420 26853 server_base.cc:1034] running on GCE node
W20260926 02:53:26.579756 26967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:26.580193 26853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:26.580876 26853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:26.582111 26853 hybrid_clock.cc:648] HybridClock initialized: now 1790391206582043 us; error 37 us; skew 500 ppm
I20260926 02:53:26.582324 26853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:26 dist-test-slave-ktng krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391206, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:26.584606 26853 init.cc:377] Logged in from keytab as oryx/127.19.185.253@KRBTEST.COM (short username oryx)
I20260926 02:53:26.588024 26853 webserver.cc:460] Webserver started at http://127.19.185.253:44423/ using document root <none> and password file <none>
I20260926 02:53:26.588428 26853 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:26.588514 26853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:26.592103 26853 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260926 02:53:26.594031 26974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:26.594472 26853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 02:53:26.594619 26853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/wal
uuid: "3bcc74bcc364436799b69b73517ab95c"
format_stamp: "Formatted at 2026-09-26 02:53:26 on dist-test-slave-ktng"
server_key: "69e6a2260a2e0c2486c0dd6353e97747"
server_key_iv: "731d8539f84db73cd91584c6a530c77e"
server_key_version: "encryptionkey@0"
I20260926 02:53:26.594779 26853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:26.620250 26853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:26.625166 26853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:26.653262 26853 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.253:34313
I20260926 02:53:26.653290 27025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.253:34313 every 8 connection(s)
I20260926 02:53:26.654232 26853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-1/data/info.pb
I20260926 02:53:26.659497 27026 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:53:26.663717 27026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: Bootstrap starting.
I20260926 02:53:26.679720 27026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:53:26.680224 27026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: Bootstrap complete.
I20260926 02:53:26.681286 27026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: NON_VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: true } }
I20260926 02:53:26.681547 27026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bcc74bcc364436799b69b73517ab95c, State: Initialized, Role: LEARNER
I20260926 02:53:26.681770 27026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: NON_VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: true } }
I20260926 02:53:26.682542 27029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: NON_VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: true } } }
I20260926 02:53:26.682823 27026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:26.682961 27029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: This master's current role is: LEARNER
I20260926 02:53:26.694492 27040 catalog_manager.cc:1259] Loaded cluster ID: c2353cf6506444608fba4eafd5e8bdb5
I20260926 02:53:26.694605 27040 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: loading cluster ID for follower catalog manager: success
I20260926 02:53:26.698128 27040 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: acquiring CA information for follower catalog manager: success
I20260926 02:53:26.700501 27040 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:53:26.815677 26848 heartbeater.cc:503] Master 127.19.185.254:45007 was elected leader, sending a full tablet report...
W20260926 02:53:26.850309 26955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:26.850577 26955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:26.850739 26955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:26.858242 26955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:26.858357 26955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:26.858446 26955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:26.858552 26955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:26.858614 26955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:53:26.869045 26955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:45007,127.19.185.253:34313,127.19.185.252:44145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.252.keytab
--principal=oryx/127.19.185.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.252:44145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.185.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:26.870450 26955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:26.871870 26955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:26.884109 27045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:26.884088 27046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:26.884680 27048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:26.884943 26955 server_base.cc:1034] running on GCE node
I20260926 02:53:26.885840 26955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:26.887223 26955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:26.888423 26955 hybrid_clock.cc:648] HybridClock initialized: now 1790391206888360 us; error 50 us; skew 500 ppm
I20260926 02:53:26.888744 26955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:26 dist-test-slave-ktng krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391206, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:26.891908 26955 init.cc:377] Logged in from keytab as oryx/127.19.185.252@KRBTEST.COM (short username oryx)
I20260926 02:53:26.893275 26955 webserver.cc:460] Webserver started at http://127.19.185.252:43319/ using document root <none> and password file <none>
I20260926 02:53:26.893903 26955 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:26.894006 26955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:26.894250 26955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:26.897461 26955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/data/instance:
uuid: "7ee830830e0d468c8175693deeaa09f7"
format_stamp: "Formatted at 2026-09-26 02:53:26 on dist-test-slave-ktng"
server_key: "3676538abb59fba2aa0f1b305d385db0"
server_key_iv: "35ef188e480869ed8cee14b5a177c5ec"
server_key_version: "encryptionkey@0"
I20260926 02:53:26.898203 26955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/wal/instance:
uuid: "7ee830830e0d468c8175693deeaa09f7"
format_stamp: "Formatted at 2026-09-26 02:53:26 on dist-test-slave-ktng"
server_key: "3676538abb59fba2aa0f1b305d385db0"
server_key_iv: "35ef188e480869ed8cee14b5a177c5ec"
server_key_version: "encryptionkey@0"
I20260926 02:53:26.903374 26955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:53:26.906635 27055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:26.908078 26955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:53:26.908250 26955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/wal
uuid: "7ee830830e0d468c8175693deeaa09f7"
format_stamp: "Formatted at 2026-09-26 02:53:26 on dist-test-slave-ktng"
server_key: "3676538abb59fba2aa0f1b305d385db0"
server_key_iv: "35ef188e480869ed8cee14b5a177c5ec"
server_key_version: "encryptionkey@0"
I20260926 02:53:26.908435 26955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:26.927736 26955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:26.932513 26955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:26.932794 26955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:26.951472 26955 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.252:44145
I20260926 02:53:26.951567 27106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.252:44145 every 8 connection(s)
I20260926 02:53:26.952869 26955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/data/info.pb
I20260926 02:53:26.956717 27107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:26.960790 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 26955
I20260926 02:53:26.964507 27107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:53:26 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391206, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.252@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
Sep 26 02:53:26 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.185.252@KRBTEST.COM
I20260926 02:53:26.988544 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.968730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46654 (local address 127.19.185.254:45007)
0926 02:53:26.968934 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:26.968940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:26.969414 (+ 474us) server_negotiation.cc:407] Connection header received
0926 02:53:26.969594 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:26.969599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:26.969680 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:26.969802 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:26.971076 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.971883 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.973751 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.974274 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.974380 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.982118 (+ 7738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:26.982123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:26.982127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:26.982186 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:26.985076 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.986043 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.986048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.986052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.986213 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.986702 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.986707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.986711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.986991 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:26.987156 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:26.987966 (+ 810us) server_negotiation.cc:299] Negotiation successful
0926 02:53:26.988196 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:53:26.989688 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.968500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:46654)
0926 02:53:26.969264 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:26.969311 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:26.969454 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:26.969855 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:26.969860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:26.970534 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:26.970913 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.970926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.972032 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.972037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.973565 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.973580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.974535 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.974541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.974660 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.978929 (+ 4269us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:26.978962 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:26.981850 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:26.985241 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:26.985255 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:26.985270 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:26.985903 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:26.986356 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:26.986361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:26.986365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:26.986579 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:26.987125 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:26.987134 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:26.987542 (+ 408us) client_negotiation.cc:769] Sending connection context
0926 02:53:26.987887 (+ 345us) client_negotiation.cc:240] Negotiation successful
0926 02:53:26.988389 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":47,"thread_start_us":349,"threads_started":1}
I20260926 02:53:26.989831 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.963268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49502 (local address 127.19.185.252:44145)
0926 02:53:26.964143 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:26.964159 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:26.964212 (+ 53us) server_negotiation.cc:407] Connection header received
0926 02:53:26.965867 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:26.965905 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:26.966490 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:26.966937 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:26.968873 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.970714 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.972169 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.973026 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.973176 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.983446 (+ 10270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:26.983458 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:26.983474 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:26.983502 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:26.986713 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.987570 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.987578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.987581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.987716 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:26.988103 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:26.988114 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:26.988116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:26.988484 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:26.989082 (+ 598us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:26.989276 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 02:53:26.989514 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":568,"thread_start_us":342,"threads_started":1}
I20260926 02:53:26.991935 27107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:53:26 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391206, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.252@KRBTEST.COM for oryx/127.19.185.253@KRBTEST.COM
I20260926 02:53:27.004874 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.992709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:34313 (local address 127.0.0.1:44312)
0926 02:53:26.992874 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:26.992892 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:26.992983 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:26.993714 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:26.993718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:26.994029 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:26.994161 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.994169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.995511 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.995514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.996716 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:26.996728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.997392 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.997396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:26.997528 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:26.998034 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:26.998050 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:26.999938 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:27.002847 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.002850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.002853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.003228 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.003624 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.003627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.003629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.003751 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.004211 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:27.004214 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:27.004327 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 02:53:27.004436 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 02:53:27.004624 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 02:53:27.004904 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:26.992875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.19.185.253:34313)
0926 02:53:26.993461 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:26.993465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:26.993481 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:53:26.993527 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:26.993530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:26.993584 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:26.993669 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:26.994300 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.995384 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.996833 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:26.997279 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:26.997371 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.000098 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:27.000102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:27.000105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:27.000143 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:27.002719 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.003401 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.003405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.003407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.003527 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.003843 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.003846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.003848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.004085 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:27.004222 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:27.004516 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 02:53:27.004658 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":328,"threads_started":1}
I20260926 02:53:27.006007 27107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:53:27 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391206, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.252@KRBTEST.COM for oryx/127.19.185.252@KRBTEST.COM
I20260926 02:53:27.018163 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.006699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:44145 (local address 127.0.0.1:49514)
0926 02:53:27.006831 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:27.006845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:27.006916 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:27.007225 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:27.007227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:27.007655 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:27.007782 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.007789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.008702 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.008705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.009506 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.009516 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.010111 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.010115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.010301 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.010810 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:27.010825 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:27.012904 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:27.016013 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.016016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.016019 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.016331 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.016684 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.016687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.016689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.016878 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.017365 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:27.017368 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:27.017464 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 02:53:27.017584 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 02:53:27.017770 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 02:53:27.018168 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.006848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.19.185.252:44145)
0926 02:53:27.006987 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:27.006991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:27.007004 (+ 13us) server_negotiation.cc:407] Connection header received
0926 02:53:27.007039 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:27.007042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:27.007090 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:27.007195 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:27.007896 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.008599 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.009618 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.009998 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.010087 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.013055 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:27.013060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:27.013063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:27.013175 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:27.015909 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.016460 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.016464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.016466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.016581 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.016997 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.017001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.017003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.017268 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:27.017372 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:27.017693 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 02:53:27.017874 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":52}
I20260926 02:53:27.022889 27107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: Bootstrap starting.
I20260926 02:53:27.026221 27107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:27.027179 27107 log.cc:824] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:27.029814 27107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: No bootstrap required, opened a new log
I20260926 02:53:27.037360 27107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } }
I20260926 02:53:27.037676 27107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:27.037811 27107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ee830830e0d468c8175693deeaa09f7, State: Initialized, Role: FOLLOWER
I20260926 02:53:27.038408 27107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } }
I20260926 02:53:27.039283 27114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } } }
I20260926 02:53:27.039698 27114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:53:27.040393 27107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 02:53:27.050004 27125 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:53:27.050143 27125 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:53:27.053913 26955 master_runner.cc:428] Detected that this master 7ee830830e0d468c8175693deeaa09f7 is joining an existing cluster
I20260926 02:53:27.054014 26955 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.185.252:44145
I20260926 02:53:27.076828 27131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.058107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:44145 (local address 127.0.0.1:49516)
0926 02:53:27.060550 (+ 2443us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:27.060577 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:27.060745 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:27.061086 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:27.061090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:27.061949 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:27.062340 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.062354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.064132 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.064150 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.065807 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.065827 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.067159 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.067165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.067363 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.068078 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:27.068106 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:27.069349 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:27.073630 (+ 4281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.073635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.073639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.074092 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.074504 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.074508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.074512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.074665 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.075356 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:27.075361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:27.075519 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 02:53:27.075691 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 02:53:27.076325 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2246,"mutex_wait_us":4,"thread_start_us":1571,"threads_started":1}
I20260926 02:53:27.077574 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.058536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49516 (local address 127.19.185.252:44145)
0926 02:53:27.058716 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:27.058723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:27.060657 (+ 1934us) server_negotiation.cc:407] Connection header received
0926 02:53:27.060869 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:27.060880 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:27.060958 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:27.061063 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:27.062498 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.063833 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.065968 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.066769 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.066889 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.069580 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:27.069586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:27.069591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:27.069636 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:27.073485 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.074238 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.074243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.074247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.074382 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.074783 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.074788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.074791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.075149 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:27.075287 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:27.076041 (+ 754us) server_negotiation.cc:299] Negotiation successful
0926 02:53:27.077170 (+ 1129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":919,"server-negotiator.queue_time_us":69}
I20260926 02:53:27.078673 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.057921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46662 (local address 127.19.185.254:45007)
0926 02:53:27.058152 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:27.058158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:27.061968 (+ 3810us) server_negotiation.cc:407] Connection header received
0926 02:53:27.062224 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:27.062232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:27.062319 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:27.062388 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:27.064034 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.065153 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.067224 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.067885 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.067990 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.070776 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:27.070782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:27.070786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:27.070845 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:27.074859 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.075749 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.075757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.075763 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.075907 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.076392 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.076396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.076399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.076725 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:27.076864 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:27.078110 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0926 02:53:27.078337 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 02:53:27.079627 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.058041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.19.185.253:34313)
0926 02:53:27.058222 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:27.058229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:27.062994 (+ 4765us) server_negotiation.cc:407] Connection header received
0926 02:53:27.063273 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:27.063279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:27.063400 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:27.063469 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:27.064997 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.066051 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.068017 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.068681 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.068788 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.071662 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:27.071669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:27.071675 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:27.071718 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:27.075464 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.076286 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.076292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.076295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.076434 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.076873 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.076878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.076881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.078325 (+ 1444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:27.078469 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:27.079032 (+ 563us) server_negotiation.cc:299] Negotiation successful
0926 02:53:27.079264 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:53:27.079911 27133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.059827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:46662)
0926 02:53:27.061868 (+ 2041us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:27.061887 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:27.062074 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:27.062622 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:27.062626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:27.063458 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:27.063842 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.063854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.065343 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.065348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.067056 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.067075 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.068249 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.068255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.068417 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.069127 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:27.069157 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:27.070551 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:27.075044 (+ 4493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.075049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.075053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.075557 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.076064 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.076069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.076073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.076272 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.077265 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:27.077270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:27.077488 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 02:53:27.079134 (+ 1646us) client_negotiation.cc:240] Negotiation successful
0926 02:53:27.079470 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1918,"thread_start_us":1317,"threads_started":1}
I20260926 02:53:27.080621 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.060285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:34313 (local address 127.0.0.1:44328)
0926 02:53:27.062889 (+ 2604us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:27.062906 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:27.063101 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:27.063675 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:27.063680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:27.064493 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:27.064815 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.064826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.066249 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.066254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.067852 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.067867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.069099 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.069106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.069330 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.069973 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:27.070007 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:27.071447 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:27.075631 (+ 4184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.075634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.075637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.076145 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.076571 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.076576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.076580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.076742 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.078544 (+ 1802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:27.078553 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:27.078751 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 02:53:27.079872 (+ 1121us) client_negotiation.cc:240] Negotiation successful
0926 02:53:27.080188 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2313,"thread_start_us":1271,"threads_started":1}
Sep 26 02:53:27 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391204, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.254@KRBTEST.COM for oryx/127.19.185.252@KRBTEST.COM
I20260926 02:53:27.095723 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 26453
I20260926 02:53:27.099404 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.088660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.19.185.253:34313)
0926 02:53:27.088819 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:27.088825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:27.090546 (+ 1721us) server_negotiation.cc:407] Connection header received
0926 02:53:27.090739 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:27.090743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:27.090809 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:27.090927 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:27.091771 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.092442 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.095680 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.097122 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.097271 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.097503 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:27.098916 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0926 02:53:27.099063 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:53:27.100198 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.088481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:34313 (local address 127.0.0.1:44332)
0926 02:53:27.090448 (+ 1967us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:27.090464 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:27.090588 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:27.091099 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:27.091103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:27.091137 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:27.091630 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.091641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.092568 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.092576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.095492 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.095512 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.098119 (+ 2607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.098130 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.098688 (+ 558us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.098753 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 02:53:27.099629 (+ 876us) client_negotiation.cc:240] Negotiation successful
0926 02:53:27.099773 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1859,"thread_start_us":1752,"threads_started":1}
I20260926 02:53:27.101293 27001 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 02:53:27.102066 27029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0987c8710d7a46eaaf9759293e32ffce. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: NON_VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: true } } }
I20260926 02:53:27.102331 27029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: This master's current role is: LEARNER
I20260926 02:53:27.102766 27134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.084936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:44145 (local address 127.0.0.1:49528)
0926 02:53:27.085498 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:27.085518 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:27.085666 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:27.086168 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:27.086171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:27.086592 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:27.086918 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.086928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.088054 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.088060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.089927 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.089948 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.090947 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.090952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.091134 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.092720 (+ 1586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:27.092749 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:27.095657 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:27.099916 (+ 4259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.099926 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.099930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.100483 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.100908 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.100915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.100921 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.101089 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.101793 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:27.101797 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:27.101947 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 02:53:27.102091 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 02:53:27.102331 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":270,"threads_started":1}
I20260926 02:53:27.103744 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.085240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.19.185.252:44145)
0926 02:53:27.085489 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:27.085493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:27.085643 (+ 150us) server_negotiation.cc:407] Connection header received
0926 02:53:27.085887 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:27.085893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:27.086031 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:27.086133 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:27.087054 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.087907 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.090106 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.090786 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.090911 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.095847 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:27.095853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:27.095858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:27.095899 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:27.099765 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.100629 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.100634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.100638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.100773 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.101222 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.101227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.101229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.101592 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:27.101732 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:27.103138 (+ 1406us) server_negotiation.cc:299] Negotiation successful
0926 02:53:27.103370 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:53:27.105062 26392 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.185.252:44145: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7ee830830e0d468c8175693deeaa09f7"
member_type: NON_VOTER
last_known_addr {
host: "127.19.185.252"
port: 44145
}
attrs {
promote: true
}
}
dest_uuid: "0987c8710d7a46eaaf9759293e32ffce"
cas_config_opid_index: 5
I20260926 02:53:27.105914 26392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: NON_VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: true } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } }
I20260926 02:53:27.114153 27001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 LEARNER]: Refusing update from remote peer 0987c8710d7a46eaaf9759293e32ffce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 02:53:27.114347 27082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 02:53:27 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.185.193@KRBTEST.COM
I20260926 02:53:27.118173 27082 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 1 FOLLOWER]: Refusing update from remote peer 0987c8710d7a46eaaf9759293e32ffce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 02:53:27.123255 27138 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7ee830830e0d468c8175693deeaa09f7 (127.19.185.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: NON_VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: true } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } } }
I20260926 02:53:27.124987 27135 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: NON_VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 02:53:27.125919 27139 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:27.128185 27138 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.185.252:44145
I20260926 02:53:27.128859 26955 master.cc:537] Master@127.19.185.252:44145 shutting down...
I20260926 02:53:27.133625 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.097510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.19.185.193:43039)
0926 02:53:27.103827 (+ 6317us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:27.103838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:27.103871 (+ 33us) server_negotiation.cc:407] Connection header received
0926 02:53:27.103942 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:27.103954 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:27.104249 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:27.104426 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:27.106140 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.107833 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.110455 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.111675 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.112689 (+ 1014us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.118631 (+ 5942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:27.118651 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:27.118678 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:27.118741 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:27.123643 (+ 4902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.127269 (+ 3626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.127278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.127282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.127462 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.128655 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.128659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.128662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.129557 (+ 895us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:27.131480 (+ 1923us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:27.132158 (+ 678us) server_negotiation.cc:299] Negotiation successful
0926 02:53:27.132476 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6135,"thread_start_us":5975,"threads_started":1}
I20260926 02:53:27.135262 26576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:27.144732 27141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: NON_VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: true } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } } }
I20260926 02:53:27.145221 27141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: This master's current role is: LEADER
W20260926 02:53:27.145725 26376 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce -> Peer 7ee830830e0d468c8175693deeaa09f7 (127.19.185.252:44145): Couldn't send request to peer 7ee830830e0d468c8175693deeaa09f7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 02:53:27.151026 27001 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7ee830830e0d468c8175693deeaa09f7 (127.19.185.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: NON_VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: true } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } } }
I20260926 02:53:27.154677 27029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: NON_VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: true } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } } }
I20260926 02:53:27.154995 27029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: This master's current role is: LEARNER
I20260926 02:53:27.155602 27140 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce: attempting to promote NON_VOTER 3bcc74bcc364436799b69b73517ab95c to VOTER
I20260926 02:53:27.156350 27140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } }
I20260926 02:53:27.158484 27001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 LEARNER]: Refusing update from remote peer 0987c8710d7a46eaaf9759293e32ffce: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 02:53:27.159018 27140 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 02:53:27.159229 26376 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce -> Peer 7ee830830e0d468c8175693deeaa09f7 (127.19.185.252:44145): Couldn't send request to peer 7ee830830e0d468c8175693deeaa09f7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 02:53:27.161231 27135 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3bcc74bcc364436799b69b73517ab95c (127.19.185.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } } }
I20260926 02:53:27.161866 27001 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3bcc74bcc364436799b69b73517ab95c (127.19.185.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } } }
I20260926 02:53:27.165349 27141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } } }
I20260926 02:53:27.165392 27138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } } }
I20260926 02:53:27.165612 27141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:27.165670 27138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:27.166198 27029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } } }
I20260926 02:53:27.166429 27029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:53:27.177675 26955 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:27.178146 26955 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:27.178323 26955 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: stopping tablet replica
I20260926 02:53:27.185585 26955 master.cc:552] Master@127.19.185.252:44145 shutdown complete.
I20260926 02:53:27.185729 26955 master_runner.cc:305] Clearing existing system tablet
I20260926 02:53:27.186192 26955 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:27.186372 26955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:27.195890 26955 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.004s
I20260926 02:53:27.197640 27147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:27.197959 26955 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:27.198242 26955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 02:53:27.198340 26955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/wal
uuid: "7ee830830e0d468c8175693deeaa09f7"
format_stamp: "Formatted at 2026-09-26 02:53:26 on dist-test-slave-ktng"
server_key: "3676538abb59fba2aa0f1b305d385db0"
server_key_iv: "35ef188e480869ed8cee14b5a177c5ec"
server_key_version: "encryptionkey@0"
I20260926 02:53:27.201736 26955 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:27.206375 26955 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 02:53:27.206561 26955 log.cc:1196] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 02:53:27.206979 26955 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: Deleting consensus metadata
I20260926 02:53:27.207227 26955 master_runner.cc:315] Copying system tablet from 127.19.185.254:45007
I20260926 02:53:27.209713 26955 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: tablet copy: Beginning tablet copy session from remote peer at address 127.19.185.254:45007
I20260926 02:53:27.222820 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.210525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.19.185.254:45007)
0926 02:53:27.210808 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:27.210817 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:27.211831 (+ 1014us) server_negotiation.cc:407] Connection header received
0926 02:53:27.212036 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:27.212041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:27.212124 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:27.212230 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:27.213006 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.213720 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.214963 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.215507 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.215643 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.217580 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:27.217586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:27.217591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:27.217631 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:27.220325 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.220930 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.220935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.220939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.221067 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.221448 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.221455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.221459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.221728 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:27.221902 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:27.222260 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 02:53:27.222465 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260926 02:53:27.222905 27153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.210502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:46674)
0926 02:53:27.211716 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:27.211736 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:27.211893 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:27.212254 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:27.212258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:27.212668 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:27.212854 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.212863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.213875 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.213880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.214814 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.214825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.215672 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.215677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.215887 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.216554 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:27.216571 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:27.217375 (+ 804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:27.220457 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.220463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.220467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.220806 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.221177 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.221182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.221186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.221337 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.221846 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:27.221851 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:27.222029 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 02:53:27.222169 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 02:53:27.222463 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"thread_start_us":966,"threads_started":1}
I20260926 02:53:27.223840 26412 tablet_copy_service.cc:140] P 0987c8710d7a46eaaf9759293e32ffce: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7ee830830e0d468c8175693deeaa09f7 ({username='oryx', principal='oryx/127.19.185.252@KRBTEST.COM'} at 127.0.0.1:46674)
I20260926 02:53:27.224010 26412 tablet_copy_service.cc:161] P 0987c8710d7a46eaaf9759293e32ffce: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7ee830830e0d468c8175693deeaa09f7 at {username='oryx', principal='oryx/127.19.185.252@KRBTEST.COM'} at 127.0.0.1:46674: session id = 7ee830830e0d468c8175693deeaa09f7-00000000000000000000000000000000
I20260926 02:53:27.226658 26412 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 02:53:27.228302 26955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 02:53:27.233568 26955 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: tablet copy: Starting download of 0 data blocks...
I20260926 02:53:27.233899 26955 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: tablet copy: Starting download of 1 WAL segments...
I20260926 02:53:27.236181 26955 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 02:53:27.239434 26412 tablet_copy_service.cc:342] P 0987c8710d7a46eaaf9759293e32ffce: Request end of tablet copy session 7ee830830e0d468c8175693deeaa09f7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.185.252@KRBTEST.COM'} at 127.0.0.1:46674
I20260926 02:53:27.239737 26412 tablet_copy_service.cc:433] P 0987c8710d7a46eaaf9759293e32ffce: ending tablet copy session 7ee830830e0d468c8175693deeaa09f7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7ee830830e0d468c8175693deeaa09f7
I20260926 02:53:27.250921 26955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:27.253916 27154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:27.259960 27155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:27.261106 27157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:27.261540 26955 server_base.cc:1034] running on GCE node
I20260926 02:53:27.261844 26955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:27.262383 26955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:27.263557 26955 hybrid_clock.cc:648] HybridClock initialized: now 1790391207263503 us; error 47 us; skew 500 ppm
I20260926 02:53:27.263801 26955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:27 dist-test-slave-ktng krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391207, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:27.265656 26955 init.cc:377] Logged in from keytab as oryx/127.19.185.252@KRBTEST.COM (short username oryx)
I20260926 02:53:27.266835 26955 webserver.cc:460] Webserver started at http://127.19.185.252:37835/ using document root <none> and password file <none>
I20260926 02:53:27.267140 26955 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:27.267252 26955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:27.269527 26955 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 02:53:27.271144 27164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:27.271551 26955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 02:53:27.271730 26955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/wal
uuid: "7ee830830e0d468c8175693deeaa09f7"
format_stamp: "Formatted at 2026-09-26 02:53:26 on dist-test-slave-ktng"
server_key: "3676538abb59fba2aa0f1b305d385db0"
server_key_iv: "35ef188e480869ed8cee14b5a177c5ec"
server_key_version: "encryptionkey@0"
I20260926 02:53:27.271937 26955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:27.292683 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 26453
I20260926 02:53:27.296988 26955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:27.300483 26955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:27.317618 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:43039
--local_ip_for_outbound_sockets=127.19.185.193
--tserver_master_addrs=127.19.185.254:45007
--webserver_port=39099
--webserver_interface=127.19.185.193
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.193.keytab
--principal=oryx/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.185.254:45007,127.19.185.253:34313
--tserver_master_addrs=127.19.185.254:45007,127.19.185.253:34313,127.19.185.252:44145 with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:53:27.351662 26955 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.252:44145
I20260926 02:53:27.351718 27216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.252:44145 every 8 connection(s)
I20260926 02:53:27.352710 26955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/master-2/data/info.pb
I20260926 02:53:27.361207 27217 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:53:27.366586 27217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: Bootstrap starting.
I20260926 02:53:27.380862 27217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:53:27.381361 27217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: Bootstrap complete.
I20260926 02:53:27.382243 27217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } }
I20260926 02:53:27.382473 27217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ee830830e0d468c8175693deeaa09f7, State: Initialized, Role: LEARNER
I20260926 02:53:27.382728 27217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } }
I20260926 02:53:27.384093 27217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:27.383872 27220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } } }
I20260926 02:53:27.384385 27220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [sys.catalog]: This master's current role is: LEARNER
I20260926 02:53:27.402302 27231 catalog_manager.cc:1259] Loaded cluster ID: c2353cf6506444608fba4eafd5e8bdb5
I20260926 02:53:27.402506 27231 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: loading cluster ID for follower catalog manager: success
I20260926 02:53:27.404973 27231 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: acquiring CA information for follower catalog manager: success
I20260926 02:53:27.406651 27231 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:53:27.675530 27232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.668702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:44145 (local address 127.0.0.1:49540)
0926 02:53:27.669318 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:27.669352 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:27.669549 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:27.670531 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:27.670536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:27.670562 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:27.670767 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.670786 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.672447 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.672453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.673589 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.673601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.674694 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.674703 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.674872 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.674905 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:53:27.675016 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 02:53:27.675182 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":496,"threads_started":1}
I20260926 02:53:27.675563 27233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.669050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.19.185.252:44145)
0926 02:53:27.670160 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:27.670168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:27.670190 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:53:27.670294 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:27.670300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:27.670385 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:27.670511 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:27.670986 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.672309 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.673734 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.674520 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.674734 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.674941 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:27.675059 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 02:53:27.675183 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":976,"thread_start_us":843,"threads_started":1}
I20260926 02:53:27.676661 27191 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 02:53:27.677289 27220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0987c8710d7a46eaaf9759293e32ffce. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } } }
I20260926 02:53:27.677608 27220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [sys.catalog]: This master's current role is: LEARNER
W20260926 02:53:27.711612 27193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:27.711889 27193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:27.711982 27193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:27.719696 27193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:27.719821 27193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:27.719939 27193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:53:27.730579 27193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.193.keytab
--principal=oryx/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:43039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=39099
--tserver_master_addrs=127.19.185.254:45007,127.19.185.253:34313,127.19.185.252:44145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:27.732020 27193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:27.733444 27193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:27.747040 27239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:27.747218 27238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:27.747195 27241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:27.748270 27193 server_base.cc:1034] running on GCE node
I20260926 02:53:27.748921 27193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:27.749919 27193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:27.751148 27193 hybrid_clock.cc:648] HybridClock initialized: now 1790391207751076 us; error 63 us; skew 500 ppm
I20260926 02:53:27.751596 27193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:27 dist-test-slave-ktng krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391207, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:27.754912 27193 init.cc:377] Logged in from keytab as oryx/127.19.185.193@KRBTEST.COM (short username oryx)
I20260926 02:53:27.759562 27193 webserver.cc:460] Webserver started at http://127.19.185.193:39099/ using document root <none> and password file <none>
I20260926 02:53:27.760552 27193 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:27.760731 27193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:27.767390 27193 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:53:27.770184 27248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:27.771714 27193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:53:27.771878 27193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "ccf68e2b2b7249a6b858e3ea0a70588b"
format_stamp: "Formatted at 2026-09-26 02:53:24 on dist-test-slave-ktng"
server_key: "7e9f7375887975271f41cbd15d2314cf"
server_key_iv: "3d904e056a048153b802bf21d883e271"
server_key_version: "encryptionkey@0"
I20260926 02:53:27.772908 27193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:27.806327 27193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:27.810963 27193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:27.811264 27193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:27.812404 27193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:27.813846 27193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:27.813928 27193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:27.814087 27193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:27.814251 27193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:27.857914 27193 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:43039
I20260926 02:53:27.857923 27360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:43039 every 8 connection(s)
I20260926 02:53:27.860280 27193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:53:27.865530 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 27193
I20260926 02:53:27.865942 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 26587
Sep 26 02:53:27 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.185.194@KRBTEST.COM
Sep 26 02:53:27 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391207, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
Sep 26 02:53:27 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391207, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.252@KRBTEST.COM
I20260926 02:53:27.957976 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.867600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.19.185.194:34167)
0926 02:53:27.921022 (+ 53422us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:27.921036 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:27.926584 (+ 5548us) server_negotiation.cc:407] Connection header received
0926 02:53:27.926706 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:27.926719 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:27.927086 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:27.927284 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:27.930537 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.932659 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.935285 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.936617 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.936764 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.945161 (+ 8397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:27.945181 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:27.945204 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:27.945269 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:27.950020 (+ 4751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.951479 (+ 1459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.951490 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.951497 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.951939 (+ 442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.952473 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.952480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.952486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.952961 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:27.954134 (+ 1173us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:27.956350 (+ 2216us) server_negotiation.cc:299] Negotiation successful
0926 02:53:27.956669 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53145,"thread_start_us":57845,"threads_started":1}
I20260926 02:53:27.960027 26710 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 26 02:53:27 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391207, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.253@KRBTEST.COM
I20260926 02:53:27.965029 27233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.863409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:35151 (local address 127.19.185.252:44145)
0926 02:53:27.863671 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:27.863682 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:27.922901 (+ 59219us) server_negotiation.cc:407] Connection header received
0926 02:53:27.939573 (+ 16672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:27.939584 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:27.939699 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:27.939786 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:27.944049 (+ 4263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.945464 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.949196 (+ 3732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.949885 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.949977 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.956453 (+ 6476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:27.956460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:27.956465 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:27.956505 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:27.960650 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.961761 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.961770 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.961777 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.961954 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.962556 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.962564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.962570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.962949 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:27.963099 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:27.964254 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0926 02:53:27.964545 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:53:27.967705 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.864064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:44145 (local address 127.19.185.193:35151)
0926 02:53:27.922702 (+ 58638us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:27.922780 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:27.939308 (+ 16528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:27.940050 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:27.940058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:27.940717 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:27.941130 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.941147 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.945712 (+ 4565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.945721 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.948048 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.948068 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.951171 (+ 3103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.951180 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.951406 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.953324 (+ 1918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:27.953350 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:27.956209 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:27.960866 (+ 4657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.960879 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.960903 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.961605 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.962117 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.962135 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.962141 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.962406 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.963216 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:27.963227 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:27.963771 (+ 544us) client_negotiation.cc:769] Sending connection context
0926 02:53:27.965342 (+ 1571us) client_negotiation.cc:240] Negotiation successful
0926 02:53:27.965985 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58232,"thread_start_us":84223,"threads_started":1}
I20260926 02:53:27.968932 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.864064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.193:48167)
0926 02:53:27.926331 (+ 62267us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:27.926358 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:27.933277 (+ 6919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:27.934898 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:27.934916 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:27.936167 (+ 1251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:27.937291 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.937323 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.941380 (+ 4057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.941388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.949010 (+ 7622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.949030 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.950316 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.950323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.950623 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.951832 (+ 1209us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:27.951864 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:27.955479 (+ 3615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:27.963371 (+ 7892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.963379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.963386 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.964096 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.964617 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.964624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.964630 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.964796 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.966089 (+ 1293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:27.966097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:27.966294 (+ 197us) client_negotiation.cc:769] Sending connection context
0926 02:53:27.966481 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 02:53:27.966850 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62064,"mutex_wait_us":382,"thread_start_us":61467,"threads_started":1}
I20260926 02:53:27.968937 27365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.864332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:48167 (local address 127.19.185.254:45007)
0926 02:53:27.930574 (+ 66242us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:27.930583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:27.930620 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:53:27.934038 (+ 3418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:27.934047 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:27.934165 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:27.934292 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:27.937624 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.939128 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.949159 (+ 10031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.949834 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.949964 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.956546 (+ 6582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:27.956552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:27.956558 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:27.956600 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:27.963144 (+ 6544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.964280 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.964289 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.964294 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.964470 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.964936 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.964944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.964950 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.965824 (+ 874us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:27.966002 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:27.968121 (+ 2119us) server_negotiation.cc:299] Negotiation successful
0926 02:53:27.968432 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66071,"thread_start_us":65954,"threads_started":1}
I20260926 02:53:27.969628 27361 heartbeater.cc:347] Connected to a master server at 127.19.185.252:44145
I20260926 02:53:27.970058 27361 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:27.970132 27362 heartbeater.cc:347] Connected to a master server at 127.19.185.254:45007
I20260926 02:53:27.970249 27362 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:27.971171 27362 heartbeater.cc:511] Master 127.19.185.254:45007 requested a full tablet report, sending...
I20260926 02:53:27.973444 26392 ts_manager.cc:194] Re-registered known tserver with Master: ccf68e2b2b7249a6b858e3ea0a70588b (127.19.185.193:43039)
I20260926 02:53:27.974773 26392 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:48167
I20260926 02:53:27.976130 27361 heartbeater.cc:511] Master 127.19.185.252:44145 requested a full tablet report, sending...
I20260926 02:53:27.978047 27181 ts_manager.cc:194] Registered new tserver with Master: ccf68e2b2b7249a6b858e3ea0a70588b (127.19.185.193:43039)
I20260926 02:53:27.981920 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.902656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:56839 (local address 127.19.185.253:34313)
0926 02:53:27.925562 (+ 22906us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:27.925571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:27.948913 (+ 23342us) server_negotiation.cc:407] Connection header received
0926 02:53:27.950270 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:27.950276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:27.950386 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:27.950477 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:27.951996 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.953271 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.955449 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.956138 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.956236 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.972367 (+ 16131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:27.972378 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:27.972385 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:27.972444 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:27.977164 (+ 4720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.978301 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.978310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.978316 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.978496 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:27.979189 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:27.979220 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:27.979226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:27.980004 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:27.980343 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:27.981337 (+ 994us) server_negotiation.cc:299] Negotiation successful
0926 02:53:27.981551 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22737,"thread_start_us":22630,"threads_started":1}
I20260926 02:53:27.981935 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:27.902052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:34313 (local address 127.19.185.193:56839)
0926 02:53:27.948796 (+ 46744us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:27.948822 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:27.949019 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:27.950745 (+ 1726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:27.950750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:27.951421 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:27.951822 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.951834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.953463 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.953469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.955178 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:27.955192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:27.957279 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:27.957286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:27.957478 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:27.958796 (+ 1318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:27.958816 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:27.971963 (+ 13147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:27.977402 (+ 5439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.977410 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.977417 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.978120 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.978676 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:27.978684 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:27.978691 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:27.978918 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:27.980512 (+ 1594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:27.980519 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:27.980897 (+ 378us) client_negotiation.cc:769] Sending connection context
0926 02:53:27.981188 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 02:53:27.981517 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46603,"thread_start_us":20063,"threads_started":1}
I20260926 02:53:27.982798 27363 heartbeater.cc:347] Connected to a master server at 127.19.185.253:34313
I20260926 02:53:27.982957 27363 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:27.983417 27363 heartbeater.cc:511] Master 127.19.185.253:34313 requested a full tablet report, sending...
I20260926 02:53:27.985342 26991 ts_manager.cc:194] Registered new tserver with Master: ccf68e2b2b7249a6b858e3ea0a70588b (127.19.185.193:43039)
I20260926 02:53:28.106523 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 26587
I20260926 02:53:28.123939 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:34167
--local_ip_for_outbound_sockets=127.19.185.194
--tserver_master_addrs=127.19.185.254:45007
--webserver_port=36443
--webserver_interface=127.19.185.194
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.194.keytab
--principal=oryx/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.185.254:45007,127.19.185.253:34313
--tserver_master_addrs=127.19.185.254:45007,127.19.185.253:34313,127.19.185.252:44145 with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:53:28.149364 27135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: NON_VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: true } } }
I20260926 02:53:28.149785 27135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:28.152033 27135 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce: attempting to promote NON_VOTER 7ee830830e0d468c8175693deeaa09f7 to VOTER
I20260926 02:53:28.152604 27135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } }
I20260926 02:53:28.154546 27191 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 1 LEARNER]: Refusing update from remote peer 0987c8710d7a46eaaf9759293e32ffce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 02:53:28.155095 27001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 FOLLOWER]: Refusing update from remote peer 0987c8710d7a46eaaf9759293e32ffce: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 02:53:28.155148 27374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 02:53:28.155810 27135 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 02:53:28.158941 27001 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7ee830830e0d468c8175693deeaa09f7 (127.19.185.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } } }
I20260926 02:53:28.159958 27191 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7ee830830e0d468c8175693deeaa09f7 (127.19.185.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } } }
I20260926 02:53:28.158305 27374 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7ee830830e0d468c8175693deeaa09f7 (127.19.185.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } } }
I20260926 02:53:28.163352 27378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } } }
I20260926 02:53:28.163734 27378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:53:28.164459 27220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } } }
I20260926 02:53:28.164710 27220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:53:28.166923 27135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } } }
I20260926 02:53:28.167217 27374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } } }
I20260926 02:53:28.167546 27135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:28.167567 27374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:28.167136 27379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0987c8710d7a46eaaf9759293e32ffce" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } } }
I20260926 02:53:28.167786 27379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce [sys.catalog]: This master's current role is: LEADER
W20260926 02:53:28.479503 27373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:28.479784 27373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:28.479961 27373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:28.488049 27373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:28.488169 27373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:28.488384 27373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:53:28.498600 27373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.194.keytab
--principal=oryx/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:34167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=36443
--tserver_master_addrs=127.19.185.254:45007,127.19.185.253:34313,127.19.185.252:44145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:28.500025 27373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:28.501507 27373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:28.515012 27384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:28.514955 27385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:28.515048 27387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:28.515957 27373 server_base.cc:1034] running on GCE node
I20260926 02:53:28.516599 27373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:28.517586 27373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:28.518821 27373 hybrid_clock.cc:648] HybridClock initialized: now 1790391208518772 us; error 41 us; skew 500 ppm
I20260926 02:53:28.519183 27373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:28 dist-test-slave-ktng krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391208, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:28.522344 27373 init.cc:377] Logged in from keytab as oryx/127.19.185.194@KRBTEST.COM (short username oryx)
I20260926 02:53:28.523835 27373 webserver.cc:460] Webserver started at http://127.19.185.194:36443/ using document root <none> and password file <none>
I20260926 02:53:28.524477 27373 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:28.524633 27373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:28.530201 27373 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:53:28.532850 27394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:28.534368 27373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:53:28.534576 27373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "9cf04e24574d4f32ac96afb67043ca5a"
format_stamp: "Formatted at 2026-09-26 02:53:25 on dist-test-slave-ktng"
server_key: "a55b4e19e5cc83c5604abb816d9089fe"
server_key_iv: "ab0f757194987440636998e8d6abeffb"
server_key_version: "encryptionkey@0"
I20260926 02:53:28.535763 27373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:28.558189 27373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:28.562572 27373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:28.562809 27373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:28.563856 27373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:28.565255 27373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:28.565344 27373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:28.565419 27373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:28.565482 27373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:28.598896 27373 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:34167
I20260926 02:53:28.598958 27506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:34167 every 8 connection(s)
I20260926 02:53:28.600488 27373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:53:28.605163 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 27373
I20260926 02:53:28.605492 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 26720
Sep 26 02:53:28 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391208, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
Sep 26 02:53:28 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391208, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.253@KRBTEST.COM
I20260926 02:53:28.643183 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:28.604122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.194:41541)
0926 02:53:28.619854 (+ 15732us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:28.619904 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:28.621693 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:28.622623 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:28.622632 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:28.623168 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:28.623844 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:28.623862 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:28.625123 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:28.625131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:28.626438 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:28.626450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:28.627220 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:28.627227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:28.627684 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:28.628701 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:28.628737 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:28.632754 (+ 4017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:28.637318 (+ 4564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:28.637328 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:28.637346 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:28.637964 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:28.638748 (+ 784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:28.638753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:28.638757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:28.639002 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:28.640223 (+ 1221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:28.640231 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:28.640699 (+ 468us) client_negotiation.cc:769] Sending connection context
0926 02:53:28.641031 (+ 332us) client_negotiation.cc:240] Negotiation successful
0926 02:53:28.641616 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15460,"thread_start_us":15247,"threads_started":1}
Sep 26 02:53:28 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391208, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.252@KRBTEST.COM
I20260926 02:53:28.643911 27514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:28.604809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:41541 (local address 127.19.185.254:45007)
0926 02:53:28.619380 (+ 14571us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:28.619386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:28.620017 (+ 631us) server_negotiation.cc:407] Connection header received
0926 02:53:28.621965 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:28.621969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:28.622057 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:28.622169 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:28.624058 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:28.624825 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:28.626581 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:28.626993 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:28.627127 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:28.632968 (+ 5841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:28.632975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:28.632980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:28.633023 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:28.637192 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:28.638255 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:28.638260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:28.638264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:28.638413 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:28.639487 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:28.639492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:28.639495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:28.639995 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:28.640130 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:28.643288 (+ 3158us) server_negotiation.cc:299] Negotiation successful
0926 02:53:28.643544 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14420,"thread_start_us":14352,"threads_started":1}
I20260926 02:53:28.645916 27508 heartbeater.cc:347] Connected to a master server at 127.19.185.254:45007
I20260926 02:53:28.646238 27508 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:28.647188 27508 heartbeater.cc:511] Master 127.19.185.254:45007 requested a full tablet report, sending...
Sep 26 02:53:28 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.185.195@KRBTEST.COM
I20260926 02:53:28.648869 26392 ts_manager.cc:194] Re-registered known tserver with Master: 9cf04e24574d4f32ac96afb67043ca5a (127.19.185.194:34167)
I20260926 02:53:28.650238 26392 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:41541
I20260926 02:53:28.651132 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:28.603380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:51613 (local address 127.19.185.252:44145)
0926 02:53:28.622368 (+ 18988us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:28.622372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:28.633659 (+ 11287us) server_negotiation.cc:407] Connection header received
0926 02:53:28.634634 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:28.634640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:28.634716 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:28.634798 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:28.636600 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:28.637742 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:28.639649 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:28.640562 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:28.640663 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:28.644592 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:28.644602 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:28.644609 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:28.644657 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:28.648159 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:28.648930 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:28.648938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:28.648942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:28.649100 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:28.649539 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:28.649547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:28.649549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:28.649903 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:28.650027 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:28.650534 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 02:53:28.650755 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18876,"thread_start_us":18726,"threads_started":1}
I20260926 02:53:28.652050 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:28.605702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:44145 (local address 127.19.185.194:51613)
0926 02:53:28.633517 (+ 27815us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:28.633540 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:28.634469 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:28.635088 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:28.635092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:28.636053 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:28.636413 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:28.636425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:28.637880 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:28.637888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:28.639487 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:28.639500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:28.641003 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:28.641010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:28.641192 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:28.641771 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:28.641791 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:28.644382 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:28.648310 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:28.648318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:28.648323 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:28.648800 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:28.649240 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:28.649245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:28.649249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:28.649405 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:28.650104 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:28.650108 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:28.650251 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 02:53:28.651346 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0926 02:53:28.651628 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27646,"mutex_wait_us":4,"thread_start_us":25604,"threads_started":1}
I20260926 02:53:28.652655 27512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:28.603476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:54249 (local address 127.19.185.253:34313)
0926 02:53:28.615633 (+ 12157us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:28.615643 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:28.631788 (+ 16145us) server_negotiation.cc:407] Connection header received
0926 02:53:28.632105 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:28.632114 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:28.632215 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:28.632311 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:28.633888 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:28.635085 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:28.638218 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:28.638864 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:28.638981 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:28.643158 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:28.643166 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:28.643173 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:28.643222 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:28.647842 (+ 4620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:28.649568 (+ 1726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:28.649573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:28.649576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:28.649702 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:28.650119 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:28.650124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:28.650127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:28.650546 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:28.651098 (+ 552us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:28.651845 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 02:53:28.652084 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11976,"thread_start_us":12165,"threads_started":1}
I20260926 02:53:28.652931 27507 heartbeater.cc:347] Connected to a master server at 127.19.185.252:44145
I20260926 02:53:28.653048 27507 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:28.653560 27507 heartbeater.cc:511] Master 127.19.185.252:44145 requested a full tablet report, sending...
I20260926 02:53:28.653970 27516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:28.605249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:34313 (local address 127.19.185.194:54249)
0926 02:53:28.631663 (+ 26414us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:28.631688 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:28.631914 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:28.632616 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:28.632624 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:28.633396 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:28.633713 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:28.633729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:28.635483 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:28.635488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:28.638015 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:28.638028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:28.639199 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:28.639207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:28.639402 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:28.640118 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:28.640139 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:28.642939 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:28.648053 (+ 5114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:28.648058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:28.648062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:28.649383 (+ 1321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:28.649835 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:28.649839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:28.649842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:28.649996 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:28.651425 (+ 1429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:28.651432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:28.651580 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 02:53:28.653134 (+ 1554us) client_negotiation.cc:240] Negotiation successful
0926 02:53:28.653452 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26240,"mutex_wait_us":108,"thread_start_us":30064,"threads_started":1}
I20260926 02:53:28.654752 27181 ts_manager.cc:194] Registered new tserver with Master: 9cf04e24574d4f32ac96afb67043ca5a (127.19.185.194:34167)
I20260926 02:53:28.654933 27509 heartbeater.cc:347] Connected to a master server at 127.19.185.253:34313
I20260926 02:53:28.655043 27509 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:28.655536 27509 heartbeater.cc:511] Master 127.19.185.253:34313 requested a full tablet report, sending...
I20260926 02:53:28.656605 26991 ts_manager.cc:194] Registered new tserver with Master: 9cf04e24574d4f32ac96afb67043ca5a (127.19.185.194:34167)
I20260926 02:53:28.658576 27518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:28.606918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.19.185.195:40521)
0926 02:53:28.631103 (+ 24185us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:28.631116 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:28.634230 (+ 3114us) server_negotiation.cc:407] Connection header received
0926 02:53:28.634434 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:28.634442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:28.634706 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:28.634844 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:28.637242 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:28.638806 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:28.640782 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:28.642275 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:28.642396 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:28.649428 (+ 7032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:28.649443 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:28.649459 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:28.649509 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:28.655542 (+ 6033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:28.656431 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:28.656440 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:28.656444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:28.656590 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:28.657023 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:28.657037 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:28.657039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:28.657416 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:28.657641 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:28.657977 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 02:53:28.658205 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23942,"thread_start_us":24611,"threads_started":1}
I20260926 02:53:28.659775 26843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:28.748688 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 26720
I20260926 02:53:28.765775 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:40521
--local_ip_for_outbound_sockets=127.19.185.195
--tserver_master_addrs=127.19.185.254:45007
--webserver_port=42783
--webserver_interface=127.19.185.195
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.195.keytab
--principal=oryx/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.185.254:45007,127.19.185.253:34313
--tserver_master_addrs=127.19.185.254:45007,127.19.185.253:34313,127.19.185.252:44145 with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:53:28.979017 27362 heartbeater.cc:503] Master 127.19.185.254:45007 was elected leader, sending a full tablet report...
W20260926 02:53:29.092135 27520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:29.092478 27520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:29.092651 27520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:29.100112 27520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:29.100255 27520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:29.100419 27520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:53:29.109998 27520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.195.keytab
--principal=oryx/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:40521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=42783
--tserver_master_addrs=127.19.185.254:45007,127.19.185.253:34313,127.19.185.252:44145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:29.111394 27520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:29.112682 27520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:29.126324 27525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:29.126255 27526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:29.126628 27528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:29.127061 27520 server_base.cc:1034] running on GCE node
I20260926 02:53:29.127764 27520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:29.128592 27520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:29.129829 27520 hybrid_clock.cc:648] HybridClock initialized: now 1790391209129784 us; error 52 us; skew 500 ppm
I20260926 02:53:29.130157 27520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:29 dist-test-slave-ktng krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391209, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:29.133838 27520 init.cc:377] Logged in from keytab as oryx/127.19.185.195@KRBTEST.COM (short username oryx)
I20260926 02:53:29.135437 27520 webserver.cc:460] Webserver started at http://127.19.185.195:42783/ using document root <none> and password file <none>
I20260926 02:53:29.136060 27520 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:29.136165 27520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:29.141510 27520 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 02:53:29.144454 27535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:29.145932 27520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:53:29.146103 27520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "855525e436144877b3db94a22a28141f"
format_stamp: "Formatted at 2026-09-26 02:53:25 on dist-test-slave-ktng"
server_key: "bf77a8478ddffedb0f0b360e180fcfc7"
server_key_iv: "739c0a15005465bfdcce000ffac9183b"
server_key_version: "encryptionkey@0"
I20260926 02:53:29.147053 27520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:29.175899 27520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:29.180454 27520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:29.180768 27520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:29.181752 27520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:29.183252 27520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:29.183424 27520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 02:53:29.183554 27520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:29.183645 27520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:29.217094 27520 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:40521
I20260926 02:53:29.217147 27647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:40521 every 8 connection(s)
I20260926 02:53:29.218528 27520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:53:29.226013 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 27520
Sep 26 02:53:29 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391209, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
Sep 26 02:53:29 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391209, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.252@KRBTEST.COM
Sep 26 02:53:29 dist-test-slave-ktng krb5kdc[26347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391209, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.253@KRBTEST.COM
I20260926 02:53:29.274119 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.221279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:40927 (local address 127.19.185.253:34313)
0926 02:53:29.241233 (+ 19954us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.241239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.241258 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:53:29.241317 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.241322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.241388 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.241680 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:29.244365 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.245459 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.259135 (+ 13676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.259878 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.259990 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.265429 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:29.265439 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:29.265446 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:29.265499 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:29.269463 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.270097 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.270103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.270107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.270261 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.271287 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.271322 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.271327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.271701 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:29.271853 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.273520 (+ 1667us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.273741 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19810,"thread_start_us":20094,"threads_started":1}
I20260926 02:53:29.275381 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.222109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:34313 (local address 127.19.185.195:40927)
0926 02:53:29.238986 (+ 16877us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:29.239011 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:29.240405 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:29.241745 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:29.241753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:29.243023 (+ 1270us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:29.244162 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.244180 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.255512 (+ 11332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.255521 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.258309 (+ 2788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.258328 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.260483 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.260494 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.260744 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.261382 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:29.261412 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:29.265171 (+ 3759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:29.269596 (+ 4425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:29.269599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:29.269601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:29.269957 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:29.270905 (+ 948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:29.270911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:29.270916 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:29.271158 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:29.272546 (+ 1388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:29.272555 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:29.273081 (+ 526us) client_negotiation.cc:769] Sending connection context
0926 02:53:29.273433 (+ 352us) client_negotiation.cc:240] Negotiation successful
0926 02:53:29.274028 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16652,"mutex_wait_us":613,"thread_start_us":15978,"threads_started":1}
I20260926 02:53:29.276728 27654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.222119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.195:40883)
0926 02:53:29.232463 (+ 10344us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:29.232516 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:29.240805 (+ 8289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:29.241566 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:29.241580 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:29.242901 (+ 1321us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:29.244174 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.244190 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.245756 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.245765 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.248523 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.248543 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.249991 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.250002 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.250354 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.251234 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:29.251272 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:29.255187 (+ 3915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:29.267213 (+ 12026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:29.267227 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:29.267255 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:29.268099 (+ 844us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:29.272674 (+ 4575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:29.272682 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:29.272689 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:29.272906 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:29.273712 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:29.273720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:29.273941 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 02:53:29.274087 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 02:53:29.274917 (+ 830us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10011,"mutex_wait_us":532,"thread_start_us":13952,"threads_started":1}
I20260926 02:53:29.277448 27650 heartbeater.cc:347] Connected to a master server at 127.19.185.253:34313
I20260926 02:53:29.277817 27650 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:29.278947 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.222421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:44145 (local address 127.19.185.195:59409)
0926 02:53:29.236598 (+ 14177us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:29.236622 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:29.240405 (+ 3783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:29.241566 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:29.241580 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:29.242895 (+ 1315us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:29.244142 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.244180 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.252055 (+ 7875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.252063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.254007 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.254025 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.257659 (+ 3634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.257666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.257850 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.258807 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:29.258836 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:29.262335 (+ 3499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:29.276094 (+ 13759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:29.276099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:29.276102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:29.276606 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:29.277061 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:29.277066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:29.277069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:29.277203 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:29.277867 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:29.277872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:29.278104 (+ 232us) client_negotiation.cc:769] Sending connection context
0926 02:53:29.278222 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 02:53:29.278508 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13997,"mutex_wait_us":61,"spinlock_wait_cycles":20224,"thread_start_us":9558,"threads_started":1}
I20260926 02:53:29.278906 27650 heartbeater.cc:511] Master 127.19.185.253:34313 requested a full tablet report, sending...
I20260926 02:53:29.279886 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.223158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:59409 (local address 127.19.185.252:44145)
0926 02:53:29.237724 (+ 14566us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.237733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.237762 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:53:29.240877 (+ 3115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.240885 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.240984 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.241091 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:29.244367 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.251086 (+ 6719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.254250 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.255033 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.255170 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.263379 (+ 8209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:29.263388 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:29.263396 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:29.263451 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:29.275930 (+ 12479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.276785 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.276790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.276793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.276931 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.277344 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.277348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.277351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.277719 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:29.279187 (+ 1468us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.279324 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.279559 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14402,"thread_start_us":14279,"threads_started":1}
I20260926 02:53:29.280787 27648 heartbeater.cc:347] Connected to a master server at 127.19.185.252:44145
I20260926 02:53:29.280907 27648 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:29.281229 26991 ts_manager.cc:194] Registered new tserver with Master: 855525e436144877b3db94a22a28141f (127.19.185.195:40521)
I20260926 02:53:29.281384 27648 heartbeater.cc:511] Master 127.19.185.252:44145 requested a full tablet report, sending...
I20260926 02:53:29.282491 27181 ts_manager.cc:194] Registered new tserver with Master: 855525e436144877b3db94a22a28141f (127.19.185.195:40521)
I20260926 02:53:29.285131 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.222846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:40883 (local address 127.19.185.254:45007)
0926 02:53:29.238216 (+ 15370us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.238226 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.238249 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:53:29.240949 (+ 2700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.240959 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.241089 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.241231 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:29.244373 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.245522 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.248816 (+ 3294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.249680 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.250893 (+ 1213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.255534 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:29.255543 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:29.255551 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:29.255607 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:29.264170 (+ 8563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.268317 (+ 4147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.268325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.268331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.268522 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.273089 (+ 4567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.273097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.273103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.273531 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:29.277474 (+ 3943us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.277739 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.279463 (+ 1724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15206,"thread_start_us":15099,"threads_started":1}
I20260926 02:53:29.286058 27649 heartbeater.cc:347] Connected to a master server at 127.19.185.254:45007
I20260926 02:53:29.286214 27649 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:29.286744 27649 heartbeater.cc:511] Master 127.19.185.254:45007 requested a full tablet report, sending...
I20260926 02:53:29.287997 26392 ts_manager.cc:194] Re-registered known tserver with Master: 855525e436144877b3db94a22a28141f (127.19.185.195:40521)
I20260926 02:53:29.289417 26392 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:40883
I20260926 02:53:29.296505 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.247591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44348 (local address 127.19.185.253:34313)
0926 02:53:29.258796 (+ 11205us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.258805 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.262631 (+ 3826us) server_negotiation.cc:407] Connection header received
0926 02:53:29.262713 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.262721 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.262819 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.262915 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:29.280290 (+ 17375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.282247 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.285051 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.285711 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.285854 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.289437 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:29.289442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:29.289446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:29.289475 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:29.292833 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.293673 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.293678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.293681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.293842 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.294320 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.294326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.294331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.294778 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:29.295090 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.296085 (+ 995us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.296239 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11028,"thread_start_us":10878,"threads_started":1}
I20260926 02:53:29.302968 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.245167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.19.185.254:45007)
0926 02:53:29.269148 (+ 23981us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.269157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.269179 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:53:29.269237 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.269245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.269333 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.269455 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:29.272567 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.274661 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.292879 (+ 18218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.293547 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.293702 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.297098 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:29.297104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:29.297109 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:29.297146 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:29.299997 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.300865 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.300869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.300871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.301105 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.301539 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.301542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.301544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.301803 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:29.302174 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.302350 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.302556 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23815,"thread_start_us":23665,"threads_started":1}
I20260926 02:53:29.307689 27667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.247931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.19.185.252:44145)
0926 02:53:29.259217 (+ 11286us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.259226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.292953 (+ 33727us) server_negotiation.cc:407] Connection header received
0926 02:53:29.293143 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.293150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.293234 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.293348 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:29.295286 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.296823 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.298350 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.298777 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.298910 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.301950 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:29.301954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:29.301958 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:29.301991 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:29.304898 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.305669 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.305674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.305677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.305832 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.306304 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.306307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.306309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.306537 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:29.306780 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.307090 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.307333 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11142,"thread_start_us":10985,"threads_started":1}
I20260926 02:53:29.307333 26392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:53:29.314920 26392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:53:29.354108 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.334610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:34167 (local address 127.0.0.1:38494)
0926 02:53:29.340771 (+ 6161us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:29.340795 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:29.340947 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:29.343319 (+ 2372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:29.343323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:29.343345 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:29.343662 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.343673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.348447 (+ 4774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.348452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.350588 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.350602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.353317 (+ 2715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.353322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.353499 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.353531 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:53:29.353609 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 02:53:29.353754 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5998,"thread_start_us":6581,"threads_started":1}
I20260926 02:53:29.354899 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.334986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38494 (local address 127.19.185.194:34167)
0926 02:53:29.340289 (+ 5303us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.340301 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.340921 (+ 620us) server_negotiation.cc:407] Connection header received
0926 02:53:29.341142 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.341174 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.341484 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.341724 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:29.344125 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.346082 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.350777 (+ 4695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.352883 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.353102 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.353296 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.354021 (+ 725us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.354399 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4990,"thread_start_us":4894,"threads_started":1}
I20260926 02:53:29.355139 27681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.335137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.19.185.193:43039)
0926 02:53:29.335812 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.335826 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.341587 (+ 5761us) server_negotiation.cc:407] Connection header received
0926 02:53:29.342083 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.342092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.342332 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.342499 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:29.343866 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.345386 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.350772 (+ 5386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.352443 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.352591 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.352768 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.354688 (+ 1920us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.354858 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":301,"threads_started":1}
I20260926 02:53:29.355942 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.334913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:43039 (local address 127.0.0.1:46924)
0926 02:53:29.341431 (+ 6518us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:29.341453 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:29.341885 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:29.342764 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:29.342769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:29.342793 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:29.343138 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.343149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.345951 (+ 2802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.345956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.348297 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.348312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.352921 (+ 4609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.352927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.353098 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.353130 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:53:29.355405 (+ 2275us) client_negotiation.cc:240] Negotiation successful
0926 02:53:29.355568 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6403,"thread_start_us":16405,"threads_started":1}
I20260926 02:53:29.360899 27296 tablet_service.cc:1467] Processing CreateTablet for tablet 9e893b790f1b4002a815e0d1dcffcdf5 (DEFAULT_TABLE table=test-table [id=abbfef9881ec4db1a280b61d0ad644cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:29.362417 27296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e893b790f1b4002a815e0d1dcffcdf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:29.363912 27442 tablet_service.cc:1467] Processing CreateTablet for tablet 9e893b790f1b4002a815e0d1dcffcdf5 (DEFAULT_TABLE table=test-table [id=abbfef9881ec4db1a280b61d0ad644cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:29.365742 27442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e893b790f1b4002a815e0d1dcffcdf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:29.366601 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.341629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:40521 (local address 127.0.0.1:36834)
0926 02:53:29.354377 (+ 12748us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:29.354396 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:29.354510 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:29.359222 (+ 4712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:29.359232 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:29.359320 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:29.359626 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.359644 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.361736 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.361742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.363804 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.363820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.365718 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.365728 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.365934 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.365971 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:53:29.366071 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 02:53:29.366218 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12652,"thread_start_us":18007,"threads_started":1}
I20260926 02:53:29.366612 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.342005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.19.185.195:40521)
0926 02:53:29.358442 (+ 16437us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.358453 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.358489 (+ 36us) server_negotiation.cc:407] Connection header received
0926 02:53:29.358597 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.358605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.358888 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.359106 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:29.359815 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.361508 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.363978 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.365542 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.365712 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.365948 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.366159 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.366310 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16232,"thread_start_us":17336,"threads_started":1}
I20260926 02:53:29.370751 27583 tablet_service.cc:1467] Processing CreateTablet for tablet 9e893b790f1b4002a815e0d1dcffcdf5 (DEFAULT_TABLE table=test-table [id=abbfef9881ec4db1a280b61d0ad644cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:29.372396 27583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e893b790f1b4002a815e0d1dcffcdf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:29.377946 27684 tablet_bootstrap.cc:492] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b: Bootstrap starting.
I20260926 02:53:29.381814 27685 tablet_bootstrap.cc:492] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a: Bootstrap starting.
I20260926 02:53:29.382332 27684 tablet_bootstrap.cc:654] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:29.383873 27684 log.cc:824] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:29.385668 27685 tablet_bootstrap.cc:654] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:29.386965 27685 log.cc:824] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:29.387542 27684 tablet_bootstrap.cc:492] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b: No bootstrap required, opened a new log
I20260926 02:53:29.387810 27684 ts_tablet_manager.cc:1397] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260926 02:53:29.392055 27687 tablet_bootstrap.cc:492] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f: Bootstrap starting.
I20260926 02:53:29.394936 27685 tablet_bootstrap.cc:492] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a: No bootstrap required, opened a new log
I20260926 02:53:29.395234 27685 ts_tablet_manager.cc:1397] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260926 02:53:29.396018 27684 raft_consensus.cc:348] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } } peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } }
I20260926 02:53:29.396456 27684 raft_consensus.cc:374] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:29.396471 27687 tablet_bootstrap.cc:654] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:29.396566 27684 raft_consensus.cc:729] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf68e2b2b7249a6b858e3ea0a70588b, State: Initialized, Role: FOLLOWER
I20260926 02:53:29.397239 27684 consensus_queue.cc:260] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } } peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } }
I20260926 02:53:29.397666 27687 log.cc:824] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:29.402204 27684 ts_tablet_manager.cc:1428] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20260926 02:53:29.405772 27685 raft_consensus.cc:348] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } } peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } }
I20260926 02:53:29.406224 27685 raft_consensus.cc:374] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:29.406347 27685 raft_consensus.cc:729] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf04e24574d4f32ac96afb67043ca5a, State: Initialized, Role: FOLLOWER
I20260926 02:53:29.406765 27687 tablet_bootstrap.cc:492] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f: No bootstrap required, opened a new log
I20260926 02:53:29.407044 27687 ts_tablet_manager.cc:1397] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260926 02:53:29.406976 27685 consensus_queue.cc:260] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } } peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } }
W20260926 02:53:29.409116 27369 tablet.cc:2369] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:53:29.412186 27508 heartbeater.cc:503] Master 127.19.185.254:45007 was elected leader, sending a full tablet report...
I20260926 02:53:29.413761 27685 ts_tablet_manager.cc:1428] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260926 02:53:29.420935 27687 raft_consensus.cc:348] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } } peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } }
I20260926 02:53:29.421437 27687 raft_consensus.cc:374] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:29.421617 27687 raft_consensus.cc:729] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 855525e436144877b3db94a22a28141f, State: Initialized, Role: FOLLOWER
I20260926 02:53:29.422284 27687 consensus_queue.cc:260] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } } peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } }
I20260926 02:53:29.423882 27649 heartbeater.cc:503] Master 127.19.185.254:45007 was elected leader, sending a full tablet report...
I20260926 02:53:29.424608 27687 ts_tablet_manager.cc:1428] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f: Time spent starting tablet: real 0.017s user 0.008s sys 0.011s
I20260926 02:53:29.467927 27689 raft_consensus.cc:482] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:53:29.468382 27689 raft_consensus.cc:504] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } } peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } }
I20260926 02:53:29.470451 27689 leader_election.cc:288] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cf04e24574d4f32ac96afb67043ca5a (127.19.185.194:34167), 855525e436144877b3db94a22a28141f (127.19.185.195:40521)
I20260926 02:53:29.482276 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.470838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:36403 (local address 127.19.185.194:34167)
0926 02:53:29.471071 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.471075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.472308 (+ 1233us) server_negotiation.cc:407] Connection header received
0926 02:53:29.472843 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.472858 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.473000 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.473122 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:29.474227 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.475502 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.479538 (+ 4036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.480949 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.481222 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.481328 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.481700 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.481825 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:53:29.482399 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.472186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:58765 (local address 127.19.185.195:40521)
0926 02:53:29.472515 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.472529 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.473274 (+ 745us) server_negotiation.cc:407] Connection header received
0926 02:53:29.473531 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.473536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.473624 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.473760 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:29.474847 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.476071 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.479724 (+ 3653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.481070 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.481250 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.481325 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.481918 (+ 593us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.482044 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:53:29.482600 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.470708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:34167 (local address 127.19.185.193:36403)
0926 02:53:29.472147 (+ 1439us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:29.472197 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:29.472548 (+ 351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:29.473218 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:29.473222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:29.473262 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:29.474054 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.474066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.475629 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.475635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.479313 (+ 3678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.479329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.481299 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.481305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.481526 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.481563 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:53:29.481675 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 02:53:29.481825 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"mutex_wait_us":30,"thread_start_us":736,"threads_started":1}
I20260926 02:53:29.483031 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.472026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:40521 (local address 127.19.185.193:58765)
0926 02:53:29.473160 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:29.473177 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:29.473342 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:29.473771 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:29.473774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:29.473792 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:29.474655 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.474668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.476272 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.476276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.479539 (+ 3263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:29.479548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.481569 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.481574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:29.481763 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.481787 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:53:29.481893 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 02:53:29.482021 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":25,"thread_start_us":742,"threads_started":1}
I20260926 02:53:29.483873 27462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e893b790f1b4002a815e0d1dcffcdf5" candidate_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf04e24574d4f32ac96afb67043ca5a" is_pre_election: true
I20260926 02:53:29.484431 27603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e893b790f1b4002a815e0d1dcffcdf5" candidate_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "855525e436144877b3db94a22a28141f" is_pre_election: true
I20260926 02:53:29.484619 27462 raft_consensus.cc:2454] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf68e2b2b7249a6b858e3ea0a70588b in term 0.
I20260926 02:53:29.485169 27603 raft_consensus.cc:2454] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf68e2b2b7249a6b858e3ea0a70588b in term 0.
I20260926 02:53:29.485663 27252 leader_election.cc:302] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf04e24574d4f32ac96afb67043ca5a, ccf68e2b2b7249a6b858e3ea0a70588b; no voters:
I20260926 02:53:29.486169 27689 raft_consensus.cc:2781] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:53:29.486414 27689 raft_consensus.cc:482] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:53:29.486596 27689 raft_consensus.cc:3037] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:29.489861 27689 raft_consensus.cc:504] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } } peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } }
W20260926 02:53:29.490360 27651 tablet.cc:2369] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:53:29.490832 27689 leader_election.cc:288] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [CANDIDATE]: Term 1 election: Requested vote from peers 9cf04e24574d4f32ac96afb67043ca5a (127.19.185.194:34167), 855525e436144877b3db94a22a28141f (127.19.185.195:40521)
I20260926 02:53:29.491461 27462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e893b790f1b4002a815e0d1dcffcdf5" candidate_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf04e24574d4f32ac96afb67043ca5a"
I20260926 02:53:29.491545 27603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e893b790f1b4002a815e0d1dcffcdf5" candidate_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "855525e436144877b3db94a22a28141f"
I20260926 02:53:29.491734 27462 raft_consensus.cc:3037] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:29.491835 27603 raft_consensus.cc:3037] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:29.495013 27603 raft_consensus.cc:2454] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf68e2b2b7249a6b858e3ea0a70588b in term 1.
I20260926 02:53:29.495051 27462 raft_consensus.cc:2454] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf68e2b2b7249a6b858e3ea0a70588b in term 1.
I20260926 02:53:29.495524 27252 leader_election.cc:302] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf04e24574d4f32ac96afb67043ca5a, ccf68e2b2b7249a6b858e3ea0a70588b; no voters:
I20260926 02:53:29.495926 27689 raft_consensus.cc:2781] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:29.496729 27689 raft_consensus.cc:686] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [term 1 LEADER]: Becoming Leader. State: Replica: ccf68e2b2b7249a6b858e3ea0a70588b, State: Running, Role: LEADER
I20260926 02:53:29.497413 27689 consensus_queue.cc:237] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } } peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } }
I20260926 02:53:29.503664 26392 catalog_manager.cc:5495] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b reported cstate change: term changed from 0 to 1, leader changed from <none> to ccf68e2b2b7249a6b858e3ea0a70588b (127.19.185.193). New cstate: current_term: 1 leader_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } health_report { overall_health: UNKNOWN } } }
I20260926 02:53:29.527050 27681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.520023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46940 (local address 127.19.185.193:43039)
0926 02:53:29.520228 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.520233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.520565 (+ 332us) server_negotiation.cc:407] Connection header received
0926 02:53:29.520805 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.520809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.520885 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.521033 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:53:29.521506 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.522469 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.524219 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.524900 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.525033 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.525800 (+ 767us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:53:29.526329 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:53:29.526566 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.526626 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.526694 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:53:29.537144 27603 raft_consensus.cc:1261] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f [term 1 FOLLOWER]: Refusing update from remote peer ccf68e2b2b7249a6b858e3ea0a70588b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:29.537160 27462 raft_consensus.cc:1261] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a [term 1 FOLLOWER]: Refusing update from remote peer ccf68e2b2b7249a6b858e3ea0a70588b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:29.538270 27695 consensus_queue.cc:1035] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:29.538820 27689 consensus_queue.cc:1035] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [LEADER]: Connected to new peer: Peer: permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:29.582270 26391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:29.582767 26391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:29.592612 26391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0987c8710d7a46eaaf9759293e32ffce: Sending DeleteTablet for 3 replicas of tablet 9e893b790f1b4002a815e0d1dcffcdf5
I20260926 02:53:29.594004 27442 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e893b790f1b4002a815e0d1dcffcdf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:29 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:38494
I20260926 02:53:29.594219 27296 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e893b790f1b4002a815e0d1dcffcdf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:29 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:46924
I20260926 02:53:29.594245 27583 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e893b790f1b4002a815e0d1dcffcdf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:29 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:36834
I20260926 02:53:29.605732 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 26360
I20260926 02:53:29.620965 27711 tablet_replica.cc:330] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f: stopping tablet replica
I20260926 02:53:29.621683 27711 raft_consensus.cc:2229] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:29.622629 27711 raft_consensus.cc:2258] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:29.624640 26422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:29.626452 27709 tablet_replica.cc:330] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a: stopping tablet replica
I20260926 02:53:29.631964 27709 raft_consensus.cc:2229] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:29.632840 27709 raft_consensus.cc:2258] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:29.635890 27709 ts_tablet_manager.cc:1905] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:29.636402 27710 tablet_replica.cc:330] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b: stopping tablet replica
I20260926 02:53:29.636605 27711 ts_tablet_manager.cc:1905] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:29.637053 27710 raft_consensus.cc:2229] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:53:29.638239 27710 raft_consensus.cc:2258] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:29.658005 27710 ts_tablet_manager.cc:1905] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:29.661106 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.602219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.19.185.253:34313)
0926 02:53:29.602556 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.602562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.622580 (+ 20018us) server_negotiation.cc:407] Connection header received
0926 02:53:29.624486 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.624495 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.624610 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.624719 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:29.629112 (+ 4393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.630674 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.633528 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.634441 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.634557 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.647540 (+ 12983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:29.647548 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:29.647553 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:29.647597 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:29.652975 (+ 5378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.654175 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.654181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.654185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.654401 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.655416 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.655421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.655437 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.655967 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:29.656283 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.660362 (+ 4079us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.660614 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260926 02:53:29.667129 27709 ts_tablet_manager.cc:1918] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:29.667448 27709 log.cc:1196] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/9e893b790f1b4002a815e0d1dcffcdf5
I20260926 02:53:29.668480 27709 ts_tablet_manager.cc:1939] T 9e893b790f1b4002a815e0d1dcffcdf5 P 9cf04e24574d4f32ac96afb67043ca5a: Deleting consensus metadata
I20260926 02:53:29.669377 27711 ts_tablet_manager.cc:1918] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:29.669718 27711 log.cc:1196] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/9e893b790f1b4002a815e0d1dcffcdf5
I20260926 02:53:29.670540 27710 ts_tablet_manager.cc:1918] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:29.670641 27711 ts_tablet_manager.cc:1939] T 9e893b790f1b4002a815e0d1dcffcdf5 P 855525e436144877b3db94a22a28141f: Deleting consensus metadata
I20260926 02:53:29.670704 27710 log.cc:1196] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/9e893b790f1b4002a815e0d1dcffcdf5
I20260926 02:53:29.671249 27710 ts_tablet_manager.cc:1939] T 9e893b790f1b4002a815e0d1dcffcdf5 P ccf68e2b2b7249a6b858e3ea0a70588b: Deleting consensus metadata
I20260926 02:53:29.692441 27667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.623414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.19.185.252:44145)
0926 02:53:29.623729 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.623738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.664126 (+ 40388us) server_negotiation.cc:407] Connection header received
0926 02:53:29.664475 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.664491 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.664679 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.664777 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:29.671699 (+ 6922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.673150 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.676766 (+ 3616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.680572 (+ 3806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.680684 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.685371 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:29.685379 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:29.685384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:29.685430 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:29.689229 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.690159 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.690164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.690169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.690310 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.690739 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.690744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.690748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.691088 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:29.691324 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.691883 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.692108 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 02:53:29.791873 26377 catalog_manager.cc:4841] TS ccf68e2b2b7249a6b858e3ea0a70588b (127.19.185.193:43039): tablet 9e893b790f1b4002a815e0d1dcffcdf5 (table test-table [id=abbfef9881ec4db1a280b61d0ad644cd]) successfully deleted
I20260926 02:53:29.792204 26379 catalog_manager.cc:4841] TS 855525e436144877b3db94a22a28141f (127.19.185.195:40521): tablet 9e893b790f1b4002a815e0d1dcffcdf5 (table test-table [id=abbfef9881ec4db1a280b61d0ad644cd]) successfully deleted
I20260926 02:53:29.792426 26379 catalog_manager.cc:4841] TS 9cf04e24574d4f32ac96afb67043ca5a (127.19.185.194:34167): tablet 9e893b790f1b4002a815e0d1dcffcdf5 (table test-table [id=abbfef9881ec4db1a280b61d0ad644cd]) successfully deleted
I20260926 02:53:29.792884 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 26360
I20260926 02:53:29.849027 27667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.819163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49556 (local address 127.19.185.252:44145)
0926 02:53:29.819409 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.819419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.830043 (+ 10624us) server_negotiation.cc:407] Connection header received
0926 02:53:29.830673 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.830684 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.830799 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.830902 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:29.835075 (+ 4173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.836284 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.838440 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.839131 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.839234 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.842944 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:29.842957 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:29.842961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:29.842998 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:29.846139 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.846899 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.846904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.846907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.847037 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.847444 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.847448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.847450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.847864 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:29.848133 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.848464 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.848665 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:53:29.853360 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:29.819154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.19.185.253:34313)
0926 02:53:29.819397 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:29.819404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:29.835388 (+ 15984us) server_negotiation.cc:407] Connection header received
0926 02:53:29.835484 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:29.835493 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:29.835612 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:29.835729 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:29.839591 (+ 3862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.840743 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.842879 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:29.843663 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:29.843800 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:29.846886 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:29.846893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:29.846897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:29.846938 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:29.850518 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.851239 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.851245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.851248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.851434 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:29.851865 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:29.851870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:29.851874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:29.852224 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:29.852458 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:29.852797 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 02:53:29.853015 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:53:30.563354 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:30.562611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.194:58629)
0926 02:53:30.563053 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:30.563168 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":219,"threads_started":1}
W20260926 02:53:30.563884 27508 heartbeater.cc:644] Failed to heartbeat to 127.19.185.254:45007 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
I20260926 02:53:30.570758 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:30.570012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.193:51349)
0926 02:53:30.570471 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:30.570592 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":229,"threads_started":1}
W20260926 02:53:30.571224 27362 heartbeater.cc:644] Failed to heartbeat to 127.19.185.254:45007 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
I20260926 02:53:30.580035 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:30.579270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.195:57557)
0926 02:53:30.579708 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:30.579815 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":240,"threads_started":1}
W20260926 02:53:30.580444 27649 heartbeater.cc:644] Failed to heartbeat to 127.19.185.254:45007 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
I20260926 02:53:31.092065 27724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0987c8710d7a46eaaf9759293e32ffce)
I20260926 02:53:31.092403 27724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } }
I20260926 02:53:31.093618 27725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0987c8710d7a46eaaf9759293e32ffce)
I20260926 02:53:31.093773 27724 leader_election.cc:288] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0987c8710d7a46eaaf9759293e32ffce (127.19.185.254:45007), 7ee830830e0d468c8175693deeaa09f7 (127.19.185.252:44145)
I20260926 02:53:31.093937 27725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } }
I20260926 02:53:31.102288 27725 leader_election.cc:288] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0987c8710d7a46eaaf9759293e32ffce (127.19.185.254:45007), 3bcc74bcc364436799b69b73517ab95c (127.19.185.253:34313)
I20260926 02:53:31.108269 27729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.096084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:36772)
0926 02:53:31.107681 (+ 11597us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:31.107937 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11403,"threads_started":1}
W20260926 02:53:31.108973 27168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
I20260926 02:53:31.109658 27727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.093956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:36770)
0926 02:53:31.109213 (+ 15257us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:31.109413 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15063,"threads_started":1}
I20260926 02:53:31.118558 27730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.111269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:36786)
0926 02:53:31.118033 (+ 6764us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:31.118275 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6555}
W20260926 02:53:31.118786 27168 leader_election.cc:334] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0987c8710d7a46eaaf9759293e32ffce (127.19.185.254:45007): Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
W20260926 02:53:31.119956 26978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
I20260926 02:53:31.120370 27729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.102581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:34313 (local address 127.0.0.1:46524)
0926 02:53:31.110005 (+ 7424us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:31.110021 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:31.110134 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:31.112184 (+ 2050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:31.112192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:31.112255 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:31.112746 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:31.112764 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:31.115000 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:31.115010 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:31.118118 (+ 3108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:31.118131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:31.119541 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:31.119550 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:31.119741 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:31.119798 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 02:53:31.119875 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 02:53:31.120023 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7335,"mutex_wait_us":70,"thread_start_us":15047,"threads_started":1}
I20260926 02:53:31.120919 27727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.094006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:44145 (local address 127.0.0.1:46012)
0926 02:53:31.109843 (+ 15837us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:31.109862 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:31.110032 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:31.110493 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:31.110499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:31.110528 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:31.110797 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:31.110811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:31.112989 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:31.112998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:31.115901 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:31.115930 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:31.119874 (+ 3944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:31.119880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:31.120093 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:31.120239 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 02:53:31.120335 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 02:53:31.120677 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15749,"mutex_wait_us":10,"thread_start_us":15662,"threads_started":1}
I20260926 02:53:31.121037 27731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.103115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46524 (local address 127.19.185.253:34313)
0926 02:53:31.111871 (+ 8756us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:31.111877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:31.111895 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:53:31.111951 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:31.111956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:31.112016 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:31.112115 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:31.113037 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:31.114783 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:31.118304 (+ 3521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:31.119286 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:31.120104 (+ 818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:31.120252 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:31.120305 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 02:53:31.120463 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":161,"server-negotiator.queue_time_us":8650,"thread_start_us":8438,"threads_started":1}
I20260926 02:53:31.121670 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.094301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.19.185.252:44145)
0926 02:53:31.109553 (+ 15252us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:31.109560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:31.110145 (+ 585us) server_negotiation.cc:407] Connection header received
0926 02:53:31.110224 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:31.110230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:31.110319 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:31.110943 (+ 624us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:31.111217 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:31.112789 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:31.116194 (+ 3405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:31.119651 (+ 3457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:31.119859 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:31.119931 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:31.121243 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0926 02:53:31.121389 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1605,"server-negotiator.queue_time_us":15143,"thread_start_us":15258,"threads_started":1}
I20260926 02:53:31.121862 27001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ee830830e0d468c8175693deeaa09f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3bcc74bcc364436799b69b73517ab95c" is_pre_election: true
I20260926 02:53:31.122157 27191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bcc74bcc364436799b69b73517ab95c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7ee830830e0d468c8175693deeaa09f7" is_pre_election: true
I20260926 02:53:31.122265 27001 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ee830830e0d468c8175693deeaa09f7 in term 1.
I20260926 02:53:31.122483 27191 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bcc74bcc364436799b69b73517ab95c in term 1.
I20260926 02:53:31.122820 27167 leader_election.cc:302] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3bcc74bcc364436799b69b73517ab95c, 7ee830830e0d468c8175693deeaa09f7; no voters: 0987c8710d7a46eaaf9759293e32ffce
I20260926 02:53:31.122973 27727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.122572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:36792)
0926 02:53:31.122715 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:31.122794 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 02:53:31.123003 26975 leader_election.cc:302] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bcc74bcc364436799b69b73517ab95c, 7ee830830e0d468c8175693deeaa09f7; no voters:
I20260926 02:53:31.123195 27725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 02:53:31.123327 27725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0987c8710d7a46eaaf9759293e32ffce)
I20260926 02:53:31.123420 27725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:53:31.123348 27724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260926 02:53:31.123235 26978 leader_election.cc:334] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0987c8710d7a46eaaf9759293e32ffce (127.19.185.254:45007): Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
I20260926 02:53:31.123525 27724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0987c8710d7a46eaaf9759293e32ffce)
I20260926 02:53:31.123625 27724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:53:31.126128 27725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } }
I20260926 02:53:31.126139 27724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } }
I20260926 02:53:31.126943 27724 leader_election.cc:288] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [CANDIDATE]: Term 2 election: Requested vote from peers 0987c8710d7a46eaaf9759293e32ffce (127.19.185.254:45007), 7ee830830e0d468c8175693deeaa09f7 (127.19.185.252:44145)
I20260926 02:53:31.127308 27729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.126878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:36812)
0926 02:53:31.127041 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:31.127137 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 02:53:31.127260 27725 leader_election.cc:288] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [CANDIDATE]: Term 2 election: Requested vote from peers 0987c8710d7a46eaaf9759293e32ffce (127.19.185.254:45007), 3bcc74bcc364436799b69b73517ab95c (127.19.185.253:34313)
I20260926 02:53:31.127516 27191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bcc74bcc364436799b69b73517ab95c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7ee830830e0d468c8175693deeaa09f7"
I20260926 02:53:31.127799 27191 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3bcc74bcc364436799b69b73517ab95c in current term 2: Already voted for candidate 7ee830830e0d468c8175693deeaa09f7 in this term.
I20260926 02:53:31.128206 27727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.127610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:36824)
0926 02:53:31.127928 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:31.128038 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260926 02:53:31.128507 27729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.128084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:36826)
0926 02:53:31.128274 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:31.128361 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 02:53:31.128578 27001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ee830830e0d468c8175693deeaa09f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3bcc74bcc364436799b69b73517ab95c"
W20260926 02:53:31.128650 27168 leader_election.cc:334] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0987c8710d7a46eaaf9759293e32ffce (127.19.185.254:45007): Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
I20260926 02:53:31.128862 27001 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ee830830e0d468c8175693deeaa09f7 in current term 2: Already voted for candidate 3bcc74bcc364436799b69b73517ab95c in this term.
I20260926 02:53:31.129284 27167 leader_election.cc:302] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ee830830e0d468c8175693deeaa09f7; no voters: 0987c8710d7a46eaaf9759293e32ffce, 3bcc74bcc364436799b69b73517ab95c
I20260926 02:53:31.129542 27725 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 02:53:31.129729 27727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.129363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:36840)
0926 02:53:31.129516 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:31.129584 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 02:53:31.129897 26978 leader_election.cc:334] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0987c8710d7a46eaaf9759293e32ffce (127.19.185.254:45007): Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
I20260926 02:53:31.130017 26978 leader_election.cc:302] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3bcc74bcc364436799b69b73517ab95c; no voters: 0987c8710d7a46eaaf9759293e32ffce, 7ee830830e0d468c8175693deeaa09f7
I20260926 02:53:31.130280 27724 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 02:53:31.565665 27734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.564706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.194:54853)
0926 02:53:31.565286 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:31.565380 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":239,"threads_started":1}
I20260926 02:53:31.572767 27735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.571924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.193:45579)
0926 02:53:31.572404 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:31.572498 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":261,"threads_started":1}
I20260926 02:53:31.581832 27736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:31.581059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.195:43709)
0926 02:53:31.581518 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:31.581653 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":258,"threads_started":1}
I20260926 02:53:32.567813 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.566812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.194:54419)
0926 02:53:32.567485 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.567597 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":343,"threads_started":1}
I20260926 02:53:32.575342 27738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.573707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.193:42137)
0926 02:53:32.575027 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.575113 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1121,"thread_start_us":1017,"threads_started":1}
I20260926 02:53:32.583406 27739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.582570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.195:58151)
0926 02:53:32.583083 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.583167 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":306,"threads_started":1}
I20260926 02:53:32.714640 27740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:53:32.714865 27740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } }
I20260926 02:53:32.715826 27740 leader_election.cc:288] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0987c8710d7a46eaaf9759293e32ffce (127.19.185.254:45007), 3bcc74bcc364436799b69b73517ab95c (127.19.185.253:34313)
I20260926 02:53:32.716509 27001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ee830830e0d468c8175693deeaa09f7" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3bcc74bcc364436799b69b73517ab95c" is_pre_election: true
I20260926 02:53:32.716744 27001 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ee830830e0d468c8175693deeaa09f7 in term 2.
I20260926 02:53:32.716792 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.715939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:37084)
0926 02:53:32.716464 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.716560 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":265,"threads_started":1}
I20260926 02:53:32.717219 27167 leader_election.cc:302] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bcc74bcc364436799b69b73517ab95c, 7ee830830e0d468c8175693deeaa09f7; no voters:
I20260926 02:53:32.717514 27740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 02:53:32.717661 27740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:53:32.717775 27740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 2 FOLLOWER]: Advancing to term 3
I20260926 02:53:32.718379 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.717997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:37094)
0926 02:53:32.718196 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.718263 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260926 02:53:32.718606 27168 leader_election.cc:334] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0987c8710d7a46eaaf9759293e32ffce (127.19.185.254:45007): Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
I20260926 02:53:32.720219 27740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } }
I20260926 02:53:32.720777 27740 leader_election.cc:288] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [CANDIDATE]: Term 3 election: Requested vote from peers 0987c8710d7a46eaaf9759293e32ffce (127.19.185.254:45007), 3bcc74bcc364436799b69b73517ab95c (127.19.185.253:34313)
I20260926 02:53:32.721312 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.720879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:37096)
0926 02:53:32.721095 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.721189 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 02:53:32.721303 27001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ee830830e0d468c8175693deeaa09f7" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3bcc74bcc364436799b69b73517ab95c"
I20260926 02:53:32.721562 27001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 2 FOLLOWER]: Advancing to term 3
I20260926 02:53:32.722244 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.721940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:37098)
0926 02:53:32.722072 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.722145 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 02:53:32.722401 27168 leader_election.cc:334] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0987c8710d7a46eaaf9759293e32ffce (127.19.185.254:45007): Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
I20260926 02:53:32.723824 27001 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ee830830e0d468c8175693deeaa09f7 in term 3.
I20260926 02:53:32.724246 27167 leader_election.cc:302] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3bcc74bcc364436799b69b73517ab95c, 7ee830830e0d468c8175693deeaa09f7; no voters: 0987c8710d7a46eaaf9759293e32ffce
I20260926 02:53:32.724568 27740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 02:53:32.725114 27740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [term 3 LEADER]: Becoming Leader. State: Replica: 7ee830830e0d468c8175693deeaa09f7, State: Running, Role: LEADER
I20260926 02:53:32.725495 27740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } }
I20260926 02:53:32.727406 27743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ee830830e0d468c8175693deeaa09f7. Latest consensus state: current_term: 3 leader_uuid: "7ee830830e0d468c8175693deeaa09f7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } } }
I20260926 02:53:32.727747 27743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:32.728637 27745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:53:32.733214 27745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:53:32.734143 27745 catalog_manager.cc:1259] Loaded cluster ID: c2353cf6506444608fba4eafd5e8bdb5
I20260926 02:53:32.734254 27745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:53:32.735515 27745 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:53:32.736310 27745 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: Loaded TSK: 0
I20260926 02:53:32.737147 27745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:53:32.819607 27180 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:53:32.826681 27180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:53:32.831215 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.830691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:37136)
0926 02:53:32.830937 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.831032 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260926 02:53:32.831223 27001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [term 3 FOLLOWER]: Refusing update from remote peer 7ee830830e0d468c8175693deeaa09f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 02:53:32.832039 27743 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 02:53:32.832399 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.832038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:37138)
0926 02:53:32.832197 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.832262 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 02:53:32.832567 27168 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 -> Peer 0987c8710d7a46eaaf9759293e32ffce (127.19.185.254:45007): Couldn't send request to peer 0987c8710d7a46eaaf9759293e32ffce. Status: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 02:53:32.835075 27747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ee830830e0d468c8175693deeaa09f7. Latest consensus state: current_term: 3 leader_uuid: "7ee830830e0d468c8175693deeaa09f7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } } }
I20260926 02:53:32.835417 27747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:53:32.836542 27743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7ee830830e0d468c8175693deeaa09f7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } } }
I20260926 02:53:32.836831 27743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:32.837044 27747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7ee830830e0d468c8175693deeaa09f7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0987c8710d7a46eaaf9759293e32ffce" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 45007 } } peers { permanent_uuid: "3bcc74bcc364436799b69b73517ab95c" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 34313 } attrs { promote: false } } peers { permanent_uuid: "7ee830830e0d468c8175693deeaa09f7" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 44145 } attrs { promote: false } } }
I20260926 02:53:32.837318 27747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bcc74bcc364436799b69b73517ab95c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:53:32.837750 27748 mvcc.cc:205] Tried to move back new op lower bound from 7333442407748861952 to 7333442407326826496. Current Snapshot: MvccSnapshot[applied={T|T < 7333442407748861952 or (T in {7333442407748861952})}]
I20260926 02:53:32.867508 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.845072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:43039 (local address 127.0.0.1:44268)
0926 02:53:32.845234 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.845253 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:32.845355 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:32.856887 (+ 11532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:32.856897 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:32.856947 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:32.857385 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:32.857402 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:32.862948 (+ 5546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:32.862954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:32.865012 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:32.865064 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:32.866641 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:32.866647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:32.866817 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:32.866853 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:53:32.866914 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 02:53:32.867052 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":70}
I20260926 02:53:32.867871 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.846043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44268 (local address 127.19.185.193:43039)
0926 02:53:32.856434 (+ 10391us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:32.856442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:32.856465 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:53:32.856549 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:32.856557 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:32.856663 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:32.860474 (+ 3811us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:32.860898 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:32.862772 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:32.865202 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:32.866422 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:32.867239 (+ 817us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:32.867347 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:32.867397 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 02:53:32.867534 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10267,"thread_start_us":10154,"threads_started":1}
I20260926 02:53:32.868111 27753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.845518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:40521 (local address 127.0.0.1:43562)
0926 02:53:32.846166 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.846188 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:32.846317 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:32.861444 (+ 15127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:32.861449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:32.861474 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:32.861869 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:32.861881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:32.863492 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:32.863498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:32.865619 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:32.865633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:32.867350 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:32.867358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:32.867544 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:32.867572 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 02:53:32.867638 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 02:53:32.867776 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":309,"threads_started":1}
I20260926 02:53:32.867089 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.846121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:34167 (local address 127.0.0.1:45252)
0926 02:53:32.859872 (+ 13751us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.859894 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:32.860058 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:32.860689 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:32.860694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:32.860719 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:32.861049 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:32.861061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:32.862598 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:32.862603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:32.864946 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:32.864960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:32.866151 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:32.866156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:32.866341 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:32.866365 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:53:32.866482 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 02:53:32.866619 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13607,"mutex_wait_us":1,"thread_start_us":13471,"threads_started":1}
I20260926 02:53:32.868216 27757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.846462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45252 (local address 127.19.185.194:34167)
0926 02:53:32.847026 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:32.847035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:32.860014 (+ 12979us) server_negotiation.cc:407] Connection header received
0926 02:53:32.860182 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:32.860189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:32.860298 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:32.860472 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:32.861254 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:32.862393 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:32.865120 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:32.865962 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:32.866150 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:32.866229 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:32.866517 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 02:53:32.866648 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":263,"threads_started":1}
I20260926 02:53:32.868988 27296 tablet_service.cc:1467] Processing CreateTablet for tablet 19f73c2477f34abd95bb0db539c9d5f0 (DEFAULT_TABLE table=test-table [id=6c8344e8b45b4383916db9c5657f64c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:32.869099 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.845677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43562 (local address 127.19.185.195:40521)
0926 02:53:32.860520 (+ 14843us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:32.860528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:32.860548 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:53:32.860615 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:32.860620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:32.860725 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:32.862018 (+ 1293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:32.862197 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:32.863338 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:32.865802 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:32.867060 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:32.868544 (+ 1484us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:32.868638 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:32.868691 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 02:53:32.868813 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14707,"thread_start_us":14604,"threads_started":1}
I20260926 02:53:32.869462 27296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f73c2477f34abd95bb0db539c9d5f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:32.869628 27442 tablet_service.cc:1467] Processing CreateTablet for tablet 19f73c2477f34abd95bb0db539c9d5f0 (DEFAULT_TABLE table=test-table [id=6c8344e8b45b4383916db9c5657f64c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:32.869866 27583 tablet_service.cc:1467] Processing CreateTablet for tablet 19f73c2477f34abd95bb0db539c9d5f0 (DEFAULT_TABLE table=test-table [id=6c8344e8b45b4383916db9c5657f64c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:32.870060 27442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f73c2477f34abd95bb0db539c9d5f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:32.870276 27583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f73c2477f34abd95bb0db539c9d5f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:32.877167 27759 tablet_bootstrap.cc:492] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f: Bootstrap starting.
I20260926 02:53:32.878165 27760 tablet_bootstrap.cc:492] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a: Bootstrap starting.
I20260926 02:53:32.878876 27758 tablet_bootstrap.cc:492] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b: Bootstrap starting.
I20260926 02:53:32.879508 27759 tablet_bootstrap.cc:654] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:32.880151 27760 tablet_bootstrap.cc:654] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:32.880761 27758 tablet_bootstrap.cc:654] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:32.881788 27759 tablet_bootstrap.cc:492] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f: No bootstrap required, opened a new log
I20260926 02:53:32.881937 27759 ts_tablet_manager.cc:1397] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260926 02:53:32.882359 27760 tablet_bootstrap.cc:492] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a: No bootstrap required, opened a new log
I20260926 02:53:32.882501 27760 ts_tablet_manager.cc:1397] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:53:32.882920 27758 tablet_bootstrap.cc:492] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b: No bootstrap required, opened a new log
I20260926 02:53:32.883067 27758 ts_tablet_manager.cc:1397] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:53:32.883332 27760 raft_consensus.cc:348] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } } peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } }
I20260926 02:53:32.883610 27760 raft_consensus.cc:374] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:32.883713 27760 raft_consensus.cc:729] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf04e24574d4f32ac96afb67043ca5a, State: Initialized, Role: FOLLOWER
I20260926 02:53:32.883998 27758 raft_consensus.cc:348] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } } peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } }
I20260926 02:53:32.883985 27760 consensus_queue.cc:260] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } } peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } }
I20260926 02:53:32.884196 27758 raft_consensus.cc:374] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:32.884254 27758 raft_consensus.cc:729] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf68e2b2b7249a6b858e3ea0a70588b, State: Initialized, Role: FOLLOWER
I20260926 02:53:32.884260 27759 raft_consensus.cc:348] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } } peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } }
I20260926 02:53:32.884603 27759 raft_consensus.cc:374] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:32.884507 27758 consensus_queue.cc:260] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } } peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } }
I20260926 02:53:32.884739 27759 raft_consensus.cc:729] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 855525e436144877b3db94a22a28141f, State: Initialized, Role: FOLLOWER
I20260926 02:53:32.885133 27759 consensus_queue.cc:260] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } } peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } }
I20260926 02:53:32.886075 27758 ts_tablet_manager.cc:1428] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 02:53:32.886667 27738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.886297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.193:58033)
0926 02:53:32.886472 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.886552 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 02:53:32.887641 27738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.887245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.193:49485)
0926 02:53:32.887420 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.887504 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 02:53:32.888998 27361 heartbeater.cc:503] Master 127.19.185.252:44145 was elected leader, sending a full tablet report...
I20260926 02:53:32.890900 27739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.890405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.195:43197)
0926 02:53:32.890640 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.890751 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260926 02:53:32.891856 27759 ts_tablet_manager.cc:1428] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20260926 02:53:32.892354 27648 heartbeater.cc:503] Master 127.19.185.252:44145 was elected leader, sending a full tablet report...
I20260926 02:53:32.892874 27760 ts_tablet_manager.cc:1428] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a: Time spent starting tablet: real 0.010s user 0.003s sys 0.006s
I20260926 02:53:32.893831 27739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.893098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.195:42101)
0926 02:53:32.893597 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.893682 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384}
I20260926 02:53:32.893885 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.893435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.194:54981)
0926 02:53:32.893640 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.893726 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 02:53:32.894963 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:32.894468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.194:40081)
0926 02:53:32.894709 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:32.894805 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260926 02:53:32.895615 27507 heartbeater.cc:503] Master 127.19.185.252:44145 was elected leader, sending a full tablet report...
W20260926 02:53:33.124099 27510 tablet.cc:2369] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:53:33.203341 27765 raft_consensus.cc:482] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:53:33.203605 27765 raft_consensus.cc:504] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } } peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } }
I20260926 02:53:33.204538 27765 leader_election.cc:288] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 855525e436144877b3db94a22a28141f (127.19.185.195:40521), 9cf04e24574d4f32ac96afb67043ca5a (127.19.185.194:34167)
I20260926 02:53:33.205155 27603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f73c2477f34abd95bb0db539c9d5f0" candidate_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "855525e436144877b3db94a22a28141f" is_pre_election: true
I20260926 02:53:33.205237 27462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f73c2477f34abd95bb0db539c9d5f0" candidate_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf04e24574d4f32ac96afb67043ca5a" is_pre_election: true
I20260926 02:53:33.205440 27603 raft_consensus.cc:2454] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf68e2b2b7249a6b858e3ea0a70588b in term 0.
I20260926 02:53:33.205498 27462 raft_consensus.cc:2454] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf68e2b2b7249a6b858e3ea0a70588b in term 0.
I20260926 02:53:33.205992 27252 leader_election.cc:302] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 855525e436144877b3db94a22a28141f, ccf68e2b2b7249a6b858e3ea0a70588b; no voters:
I20260926 02:53:33.206280 27765 raft_consensus.cc:2781] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:53:33.206378 27765 raft_consensus.cc:482] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:53:33.206503 27765 raft_consensus.cc:3037] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:33.208607 27765 raft_consensus.cc:504] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } } peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } }
I20260926 02:53:33.209076 27765 leader_election.cc:288] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [CANDIDATE]: Term 1 election: Requested vote from peers 855525e436144877b3db94a22a28141f (127.19.185.195:40521), 9cf04e24574d4f32ac96afb67043ca5a (127.19.185.194:34167)
I20260926 02:53:33.209580 27603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f73c2477f34abd95bb0db539c9d5f0" candidate_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "855525e436144877b3db94a22a28141f"
I20260926 02:53:33.209719 27462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f73c2477f34abd95bb0db539c9d5f0" candidate_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf04e24574d4f32ac96afb67043ca5a"
I20260926 02:53:33.209779 27603 raft_consensus.cc:3037] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:33.209904 27462 raft_consensus.cc:3037] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:33.212150 27462 raft_consensus.cc:2454] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf68e2b2b7249a6b858e3ea0a70588b in term 1.
I20260926 02:53:33.212268 27603 raft_consensus.cc:2454] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf68e2b2b7249a6b858e3ea0a70588b in term 1.
I20260926 02:53:33.212560 27252 leader_election.cc:302] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf04e24574d4f32ac96afb67043ca5a, ccf68e2b2b7249a6b858e3ea0a70588b; no voters:
I20260926 02:53:33.212841 27765 raft_consensus.cc:2781] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:33.213286 27765 raft_consensus.cc:686] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [term 1 LEADER]: Becoming Leader. State: Replica: ccf68e2b2b7249a6b858e3ea0a70588b, State: Running, Role: LEADER
I20260926 02:53:33.213696 27765 consensus_queue.cc:237] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } } peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } }
I20260926 02:53:33.214489 27738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.214006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.193:58537)
0926 02:53:33.214201 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:33.214292 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 02:53:33.216711 27180 catalog_manager.cc:5495] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b reported cstate change: term changed from 0 to 1, leader changed from <none> to ccf68e2b2b7249a6b858e3ea0a70588b (127.19.185.193). New cstate: current_term: 1 leader_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccf68e2b2b7249a6b858e3ea0a70588b" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 } health_report { overall_health: UNKNOWN } } }
I20260926 02:53:33.310060 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.305367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44278 (local address 127.19.185.193:43039)
0926 02:53:33.305576 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:33.305582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:33.305604 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:53:33.305825 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:33.305830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:33.305902 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:33.306001 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:53:33.306366 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:33.307073 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:33.308448 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:33.308853 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:33.308995 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:33.309363 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:53:33.309475 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:53:33.309577 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:33.309725 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 02:53:33.309804 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:53:33.313112 27603 raft_consensus.cc:1261] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f [term 1 FOLLOWER]: Refusing update from remote peer ccf68e2b2b7249a6b858e3ea0a70588b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:33.313175 27462 raft_consensus.cc:1261] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a [term 1 FOLLOWER]: Refusing update from remote peer ccf68e2b2b7249a6b858e3ea0a70588b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:33.313773 27767 consensus_queue.cc:1035] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [LEADER]: Connected to new peer: Peer: permanent_uuid: "855525e436144877b3db94a22a28141f" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:33.314147 27765 consensus_queue.cc:1035] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf04e24574d4f32ac96afb67043ca5a" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:33.316579 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.316134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.194:38819)
0926 02:53:33.316295 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:33.316387 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 02:53:33.319748 27739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.317276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.195:51779)
0926 02:53:33.317512 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:33.317602 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260926 02:53:33.322199 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.321656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.194:51593)
0926 02:53:33.321921 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:33.322013 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 02:53:33.324537 27739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.323980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.195:45759)
0926 02:53:33.324294 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:33.324379 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
I20260926 02:53:33.325881 27738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.325393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.193:58305)
0926 02:53:33.325604 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:33.325690 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 02:53:33.332962 27772 mvcc.cc:205] Tried to move back new op lower bound from 7333442409724510208 to 7333442409325658112. Current Snapshot: MvccSnapshot[applied={T|T < 7333442409724510208 or (T in {7333442409724510208})}]
I20260926 02:53:33.334733 27179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:33.335065 27179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:33.341293 27738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.339990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.19.185.193:57325)
0926 02:53:33.341043 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:33.341138 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":919}
I20260926 02:53:33.342558 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.342093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:37150)
0926 02:53:33.342314 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:33.342406 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260926 02:53:33.344218 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.343767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:37154)
0926 02:53:33.343969 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:33.344085 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 02:53:33.344568 27179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7: Sending DeleteTablet for 3 replicas of tablet 19f73c2477f34abd95bb0db539c9d5f0
I20260926 02:53:33.345378 27296 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f73c2477f34abd95bb0db539c9d5f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:33 UTC) from {username='oryx', principal='oryx/127.19.185.252@KRBTEST.COM'} at 127.0.0.1:44268
I20260926 02:53:33.345811 27442 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f73c2477f34abd95bb0db539c9d5f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:33 UTC) from {username='oryx', principal='oryx/127.19.185.252@KRBTEST.COM'} at 127.0.0.1:45252
I20260926 02:53:33.346495 27441 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f73c2477f34abd95bb0db539c9d5f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:33 UTC) from {username='oryx', principal='oryx/127.19.185.252@KRBTEST.COM'} at 127.0.0.1:45252
I20260926 02:53:33.346904 27780 tablet_replica.cc:330] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b: stopping tablet replica
I20260926 02:53:33.346926 27583 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f73c2477f34abd95bb0db539c9d5f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:33 UTC) from {username='oryx', principal='oryx/127.19.185.252@KRBTEST.COM'} at 127.0.0.1:43562
I20260926 02:53:33.347033 27780 raft_consensus.cc:2229] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:53:33.347671 27782 tablet_replica.cc:330] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f: stopping tablet replica
I20260926 02:53:33.347692 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 27193
I20260926 02:53:33.347837 27782 raft_consensus.cc:2229] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:33.348106 27782 raft_consensus.cc:2258] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:33.348397 27781 tablet_replica.cc:330] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a: stopping tablet replica
I20260926 02:53:33.348636 27781 raft_consensus.cc:2229] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:33.348941 27781 raft_consensus.cc:2258] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:33.350173 27780 raft_consensus.cc:2258] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:33.351375 27780 ts_tablet_manager.cc:1905] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:33.355480 27782 ts_tablet_manager.cc:1905] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:33.357666 27781 ts_tablet_manager.cc:1905] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:33.358614 27780 ts_tablet_manager.cc:1918] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:33.358822 27780 log.cc:1196] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/19f73c2477f34abd95bb0db539c9d5f0
I20260926 02:53:33.359392 27780 ts_tablet_manager.cc:1939] T 19f73c2477f34abd95bb0db539c9d5f0 P ccf68e2b2b7249a6b858e3ea0a70588b: Deleting consensus metadata
I20260926 02:53:33.360760 27166 catalog_manager.cc:4841] TS ccf68e2b2b7249a6b858e3ea0a70588b (127.19.185.193:43039): tablet 19f73c2477f34abd95bb0db539c9d5f0 (table test-table [id=6c8344e8b45b4383916db9c5657f64c3]) successfully deleted
I20260926 02:53:33.364254 27781 ts_tablet_manager.cc:1918] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:33.364454 27781 log.cc:1196] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/19f73c2477f34abd95bb0db539c9d5f0
I20260926 02:53:33.364945 27781 ts_tablet_manager.cc:1939] T 19f73c2477f34abd95bb0db539c9d5f0 P 9cf04e24574d4f32ac96afb67043ca5a: Deleting consensus metadata
I20260926 02:53:33.366158 27168 catalog_manager.cc:4841] TS 9cf04e24574d4f32ac96afb67043ca5a (127.19.185.194:34167): tablet 19f73c2477f34abd95bb0db539c9d5f0 (table test-table [id=6c8344e8b45b4383916db9c5657f64c3]) successfully deleted
I20260926 02:53:33.366519 27782 ts_tablet_manager.cc:1918] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:33.366701 27782 log.cc:1196] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/19f73c2477f34abd95bb0db539c9d5f0
W20260926 02:53:33.366842 27168 catalog_manager.cc:4808] TS 9cf04e24574d4f32ac96afb67043ca5a (127.19.185.194:34167): delete failed for tablet 19f73c2477f34abd95bb0db539c9d5f0 because the tablet was not found. No further retry: Not found: Tablet not found: 19f73c2477f34abd95bb0db539c9d5f0
I20260926 02:53:33.367172 27782 ts_tablet_manager.cc:1939] T 19f73c2477f34abd95bb0db539c9d5f0 P 855525e436144877b3db94a22a28141f: Deleting consensus metadata
I20260926 02:53:33.367506 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.349211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.19.185.193:43039)
0926 02:53:33.349379 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:33.349386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:33.349859 (+ 473us) server_negotiation.cc:407] Connection header received
0926 02:53:33.350171 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:33.350175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:33.350283 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:33.350358 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:33.352312 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:33.353405 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:33.355233 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:33.355958 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:33.356062 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:33.359762 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:33.359781 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:33.359801 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:33.359843 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:33.363957 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:33.364864 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:33.364873 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:33.364877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:33.365031 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:33.365480 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:33.365485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:33.365489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:33.365915 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:33.366164 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:33.366793 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 02:53:33.367052 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:53:33.368368 27168 catalog_manager.cc:4841] TS 855525e436144877b3db94a22a28141f (127.19.185.195:40521): tablet 19f73c2477f34abd95bb0db539c9d5f0 (table test-table [id=6c8344e8b45b4383916db9c5657f64c3]) successfully deleted
I20260926 02:53:33.368556 27356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:33.489502 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 27193
I20260926 02:53:33.511276 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 27373
I20260926 02:53:33.525094 27785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.512210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.19.185.194:34167)
0926 02:53:33.512699 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:33.512703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:33.512720 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:53:33.512775 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:33.512778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:33.512843 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:33.513011 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:33.514281 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:33.515010 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:33.516353 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:33.516788 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:33.516889 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:33.519347 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:33.519367 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:33.519391 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:33.519436 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:33.522446 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:33.523081 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:33.523088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:33.523090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:33.523250 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:33.523687 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:33.523691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:33.523693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:33.524064 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:33.524292 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:33.524527 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 02:53:33.524686 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":284,"threads_started":1}
I20260926 02:53:33.525859 27502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:33.624401 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 27373
I20260926 02:53:33.641626 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 27520
I20260926 02:53:33.655851 27787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.642335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.19.185.195:40521)
0926 02:53:33.642814 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:33.642819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:33.642836 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:53:33.642891 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:33.642895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:33.642962 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:33.643062 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:33.644533 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:33.645251 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:33.646675 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:33.647112 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:33.647216 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:33.649889 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:33.649903 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:33.649919 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:33.649952 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:33.652900 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:33.653631 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:33.653641 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:33.653645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:33.653823 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:33.654235 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:33.654240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:33.654242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:33.654691 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:33.654970 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:33.655210 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 02:53:33.655501 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":278,"threads_started":1}
I20260926 02:53:33.656600 27643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:33.757303 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 27520
I20260926 02:53:33.775058 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 26853
I20260926 02:53:33.776230 27021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:33.863816 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.861317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:37156)
0926 02:53:33.863572 (+ 2255us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:33.863660 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2139,"thread_start_us":2027,"threads_started":1}
I20260926 02:53:33.865103 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.864760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:45007 (local address 127.0.0.1:37172)
0926 02:53:33.864922 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:33.864990 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 02:53:33.892482 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 26853
W20260926 02:53:33.913028 27167 connection.cc:535] client connection to 127.19.185.253:34313 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 02:53:33.913410 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 26955
I20260926 02:53:33.914333 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:33.913832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:34313 (local address 127.0.0.1:46538)
0926 02:53:33.914051 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:33.914139 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 02:53:33.914513 27212 generic_service.cc:196] Checking for leaks (request via RPC)
W20260926 02:53:33.914515 27167 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7ee830830e0d468c8175693deeaa09f7 -> Peer 3bcc74bcc364436799b69b73517ab95c (127.19.185.253:34313): Couldn't send request to peer 3bcc74bcc364436799b69b73517ab95c. Status: Network error: Client connection negotiation failed: client connection to 127.19.185.253:34313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 02:53:34.036218 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 26955
2026-09-26T02:53:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11623 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:53:34 dist-test-slave-ktng krb5kdc[27796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:53:34 dist-test-slave-ktng krb5kdc[27796](info): set up 2 sockets
Sep 26 02:53:34 dist-test-slave-ktng krb5kdc[27796](info): commencing operation
krb5kdc: starting...
W20260926 02:53:35.129134 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:53:35 dist-test-slave-ktng krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:53:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.254.keytab.
Entry for principal oryx/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.254.keytab.
I20260926 02:53:35.360069 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:37991
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.254.keytab
--principal=oryx/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:41629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:37991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:35.685953 27809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:35.686250 27809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:35.686403 27809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:35.693562 27809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:35.693665 27809 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 02:53:35.693750 27809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:35.693862 27809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:35.693943 27809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:35.694048 27809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:53:35.703873 27809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:37991
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.254.keytab
--principal=oryx/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:37991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:35.705204 27809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:35.706636 27809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:35.720254 27815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:35.720295 27814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:35.720295 27817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:35.720608 27809 server_base.cc:1034] running on GCE node
I20260926 02:53:35.721761 27809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:35.723176 27809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:35.724424 27809 hybrid_clock.cc:648] HybridClock initialized: now 1790391215724351 us; error 46 us; skew 500 ppm
I20260926 02:53:35.724748 27809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:35 dist-test-slave-ktng krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391215, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:35.727923 27809 init.cc:377] Logged in from keytab as oryx/127.19.185.254@KRBTEST.COM (short username oryx)
I20260926 02:53:35.729346 27809 webserver.cc:460] Webserver started at http://127.19.185.254:43561/ using document root <none> and password file <none>
I20260926 02:53:35.730021 27809 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:35.730106 27809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:35.730366 27809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:35.733384 27809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "c907464d33f54617ae3856d4f75cab9f"
format_stamp: "Formatted at 2026-09-26 02:53:35 on dist-test-slave-ktng"
server_key: "aa5a4cd1eb2a42647762a9fe4579c9bc"
server_key_iv: "c979d18894c1654d3aea18cda41ba4ba"
server_key_version: "encryptionkey@0"
I20260926 02:53:35.734156 27809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "c907464d33f54617ae3856d4f75cab9f"
format_stamp: "Formatted at 2026-09-26 02:53:35 on dist-test-slave-ktng"
server_key: "aa5a4cd1eb2a42647762a9fe4579c9bc"
server_key_iv: "c979d18894c1654d3aea18cda41ba4ba"
server_key_version: "encryptionkey@0"
I20260926 02:53:35.739848 27809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 02:53:35.743443 27824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:35.744849 27809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:53:35.745072 27809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "c907464d33f54617ae3856d4f75cab9f"
format_stamp: "Formatted at 2026-09-26 02:53:35 on dist-test-slave-ktng"
server_key: "aa5a4cd1eb2a42647762a9fe4579c9bc"
server_key_iv: "c979d18894c1654d3aea18cda41ba4ba"
server_key_version: "encryptionkey@0"
I20260926 02:53:35.745306 27809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:35.768735 27809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:35.773336 27809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:35.773619 27809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:35.796443 27809 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:37991
I20260926 02:53:35.796471 27885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:37991 every 8 connection(s)
I20260926 02:53:35.797896 27809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:53:35.801750 27886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:35.802484 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 27809
I20260926 02:53:35.802696 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:53:35.803570 20199 external_mini_cluster.cc:1428] Setting key 807066fbc100684e5d4883d46f53e396
I20260926 02:53:35.816412 27886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f: Bootstrap starting.
Sep 26 02:53:35 dist-test-slave-ktng krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
I20260926 02:53:35.820206 27886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:35.821311 27886 log.cc:824] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:35.824282 27886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f: No bootstrap required, opened a new log
I20260926 02:53:35.828094 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:35.805852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34282 (local address 127.19.185.254:37991)
0926 02:53:35.806761 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:35.806777 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:35.806822 (+ 45us) server_negotiation.cc:407] Connection header received
0926 02:53:35.808532 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:35.808555 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:35.808990 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:35.809322 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:35.811122 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:35.812542 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:35.814364 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:35.815043 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:35.815187 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:35.819839 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:35.819847 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:35.819862 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:35.819901 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:35.823496 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:35.824284 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:35.824291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:35.824297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:35.824542 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:35.824973 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:35.824983 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:35.824987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:35.825527 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:35.825901 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:35.826381 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 02:53:35.826893 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":411,"threads_started":1}
I20260926 02:53:35.832957 27886 raft_consensus.cc:348] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c907464d33f54617ae3856d4f75cab9f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37991 } }
I20260926 02:53:35.833372 27886 raft_consensus.cc:374] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:35.833479 27886 raft_consensus.cc:729] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c907464d33f54617ae3856d4f75cab9f, State: Initialized, Role: FOLLOWER
I20260926 02:53:35.834086 27886 consensus_queue.cc:260] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c907464d33f54617ae3856d4f75cab9f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37991 } }
I20260926 02:53:35.834299 27886 raft_consensus.cc:388] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:53:35.834452 27886 raft_consensus.cc:482] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:53:35.834630 27886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:35.836863 27886 raft_consensus.cc:504] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c907464d33f54617ae3856d4f75cab9f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37991 } }
I20260926 02:53:35.837392 27886 leader_election.cc:302] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c907464d33f54617ae3856d4f75cab9f; no voters:
I20260926 02:53:35.838066 27886 leader_election.cc:288] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:53:35.838244 27891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:35.839079 27891 raft_consensus.cc:686] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [term 1 LEADER]: Becoming Leader. State: Replica: c907464d33f54617ae3856d4f75cab9f, State: Running, Role: LEADER
I20260926 02:53:35.839816 27891 consensus_queue.cc:237] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c907464d33f54617ae3856d4f75cab9f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37991 } }
I20260926 02:53:35.840464 27886 sys_catalog.cc:564] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:35.843086 27893 sys_catalog.cc:455] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c907464d33f54617ae3856d4f75cab9f. Latest consensus state: current_term: 1 leader_uuid: "c907464d33f54617ae3856d4f75cab9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c907464d33f54617ae3856d4f75cab9f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37991 } } }
I20260926 02:53:35.843595 27893 sys_catalog.cc:458] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:35.843849 27892 sys_catalog.cc:455] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c907464d33f54617ae3856d4f75cab9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c907464d33f54617ae3856d4f75cab9f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37991 } } }
I20260926 02:53:35.844126 27892 sys_catalog.cc:458] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:35.848392 27901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:53:35.853452 27901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:53:35.863471 27901 catalog_manager.cc:1347] Generated new cluster ID: 34ea70d9832a4057a04b1eb33043e914
I20260926 02:53:35.863588 27901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:53:35.883210 27901 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:53:35.884577 27901 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:53:35.890688 27901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f: Generated new TSK 0
I20260926 02:53:35.891681 27901 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.193.keytab.
Entry for principal oryx/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.193.keytab.
I20260926 02:53:35.956175 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37991
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.193.keytab
--principal=oryx/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:41629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:36.288837 27912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:36.289165 27912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:36.289333 27912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:36.296653 27912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:36.296808 27912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:36.296947 27912 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:53:36.297082 27912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:53:36.306922 27912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.193.keytab
--principal=oryx/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.185.254:37991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:36.308355 27912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:36.309731 27912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:36.323378 27917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:36.323310 27920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:36.323405 27918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:36.324693 27912 server_base.cc:1034] running on GCE node
I20260926 02:53:36.325318 27912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:36.326357 27912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:36.327625 27912 hybrid_clock.cc:648] HybridClock initialized: now 1790391216327553 us; error 36 us; skew 500 ppm
I20260926 02:53:36.327940 27912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:36 dist-test-slave-ktng krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391216, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:36.331331 27912 init.cc:377] Logged in from keytab as oryx/127.19.185.193@KRBTEST.COM (short username oryx)
I20260926 02:53:36.332850 27912 webserver.cc:460] Webserver started at http://127.19.185.193:39597/ using document root <none> and password file <none>
I20260926 02:53:36.333567 27912 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:36.333701 27912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:36.334038 27912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:36.337011 27912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "05960a65b34b4cd7ad9f4a61855240d3"
format_stamp: "Formatted at 2026-09-26 02:53:36 on dist-test-slave-ktng"
server_key: "84e347ab76c9c138d20bdb87a284eccd"
server_key_iv: "ac3f5dfcc8512d9f1ae1efefa432dc51"
server_key_version: "encryptionkey@0"
I20260926 02:53:36.337816 27912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "05960a65b34b4cd7ad9f4a61855240d3"
format_stamp: "Formatted at 2026-09-26 02:53:36 on dist-test-slave-ktng"
server_key: "84e347ab76c9c138d20bdb87a284eccd"
server_key_iv: "ac3f5dfcc8512d9f1ae1efefa432dc51"
server_key_version: "encryptionkey@0"
I20260926 02:53:36.343552 27912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:53:36.347222 27927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:36.349195 27912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:53:36.349375 27912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "05960a65b34b4cd7ad9f4a61855240d3"
format_stamp: "Formatted at 2026-09-26 02:53:36 on dist-test-slave-ktng"
server_key: "84e347ab76c9c138d20bdb87a284eccd"
server_key_iv: "ac3f5dfcc8512d9f1ae1efefa432dc51"
server_key_version: "encryptionkey@0"
I20260926 02:53:36.349617 27912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:36.370285 27912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:36.374819 27912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:36.375253 27912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:36.378156 27912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:36.378260 27912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:36.378398 27912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:36.378544 27912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 26 02:53:36 dist-test-slave-ktng krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391216, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
I20260926 02:53:36.405798 27937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:36.379415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:37991 (local address 127.19.185.193:56297)
0926 02:53:36.380385 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:36.380430 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:36.382418 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:36.383228 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:36.383240 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:36.384210 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:36.385218 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:36.385253 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.386860 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.386875 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:36.389432 (+ 2557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:36.389465 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.390926 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.390936 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:36.391194 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:36.392175 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:36.392203 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:36.395897 (+ 3694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:36.400725 (+ 4828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:36.400735 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:36.400754 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:36.401302 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:36.401809 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:36.401813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:36.401816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:36.402116 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:36.402896 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:36.402902 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:36.403486 (+ 584us) client_negotiation.cc:769] Sending connection context
0926 02:53:36.403817 (+ 331us) client_negotiation.cc:240] Negotiation successful
0926 02:53:36.404404 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":449,"threads_started":1}
I20260926 02:53:36.406093 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:36.378593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:56297 (local address 127.19.185.254:37991)
0926 02:53:36.379132 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:36.379143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:36.380565 (+ 1422us) server_negotiation.cc:407] Connection header received
0926 02:53:36.382725 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:36.382732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:36.382824 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:36.382968 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:36.385431 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.386590 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.389652 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.390610 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.390782 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:36.396206 (+ 5424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:36.396216 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:36.396224 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:36.396273 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:36.400539 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:36.401464 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:36.401473 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:36.401479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:36.401658 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:36.402253 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:36.402260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:36.402266 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:36.402740 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:36.402922 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:36.405236 (+ 2314us) server_negotiation.cc:299] Negotiation successful
0926 02:53:36.405536 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":264,"threads_started":1}
I20260926 02:53:36.426112 27912 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:39795
I20260926 02:53:36.426131 28047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:39795 every 8 connection(s)
I20260926 02:53:36.427474 27912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:53:36.428913 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:36.413533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:45207 (local address 127.19.185.254:37991)
0926 02:53:36.413743 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:36.413749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:36.414692 (+ 943us) server_negotiation.cc:407] Connection header received
0926 02:53:36.414899 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:36.414904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:36.414995 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:36.415113 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:36.416850 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.417615 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.419773 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.420403 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.420547 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:36.423182 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:36.423187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:36.423190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:36.423248 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:36.426166 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:36.426858 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:36.426865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:36.426869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:36.427011 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:36.427403 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:36.427409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:36.427412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:36.427751 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:36.427913 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:36.428329 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 02:53:36.428546 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:53:36.429032 28016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:36.413639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:37991 (local address 127.19.185.193:45207)
0926 02:53:36.414556 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:36.414580 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:36.414744 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:36.415342 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:36.415350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:36.416197 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:36.416690 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:36.416706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.417768 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.417773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:36.419618 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:36.419640 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.420612 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.420620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:36.420782 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:36.421687 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:36.421704 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:36.422986 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:36.426351 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:36.426357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:36.426362 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:36.426742 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:36.427128 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:36.427132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:36.427136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:36.427251 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:36.427865 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:36.427870 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:36.428060 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 02:53:36.428231 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 02:53:36.428568 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":441,"threads_started":1}
I20260926 02:53:36.430817 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 27912
I20260926 02:53:36.430976 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:53:36.431685 20199 external_mini_cluster.cc:1428] Setting key aec96d815ce3eb12f821f1ad88aec6e7
I20260926 02:53:36.430078 28048 heartbeater.cc:347] Connected to a master server at 127.19.185.254:37991
I20260926 02:53:36.432464 28048 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:36.433261 28048 heartbeater.cc:511] Master 127.19.185.254:37991 requested a full tablet report, sending...
I20260926 02:53:36.435510 27841 ts_manager.cc:194] Registered new tserver with Master: 05960a65b34b4cd7ad9f4a61855240d3 (127.19.185.193:39795)
I20260926 02:53:36.437067 27841 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:56297
WARNING: no policy specified for oryx/127.19.185.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.185.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.194.keytab.
Entry for principal oryx/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.194.keytab.
I20260926 02:53:36.487418 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37991
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.194.keytab
--principal=oryx/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:41629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:36.819044 28052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:36.819423 28052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:36.819572 28052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:36.826993 28052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:36.827098 28052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:36.827188 28052 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:53:36.827268 28052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:53:36.836995 28052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.194.keytab
--principal=oryx/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.185.254:37991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:36.838215 28052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:36.839615 28052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:36.853322 28057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:36.853544 28058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:36.853763 28060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:36.854542 28052 server_base.cc:1034] running on GCE node
I20260926 02:53:36.855218 28052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:36.856227 28052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:36.857451 28052 hybrid_clock.cc:648] HybridClock initialized: now 1790391216857371 us; error 57 us; skew 500 ppm
I20260926 02:53:36.857774 28052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:36 dist-test-slave-ktng krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391216, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:36.861023 28052 init.cc:377] Logged in from keytab as oryx/127.19.185.194@KRBTEST.COM (short username oryx)
I20260926 02:53:36.862463 28052 webserver.cc:460] Webserver started at http://127.19.185.194:38143/ using document root <none> and password file <none>
I20260926 02:53:36.863200 28052 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:36.863359 28052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:36.863704 28052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:36.866600 28052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "484aa4de22bf4443ad1cc14ec37366be"
format_stamp: "Formatted at 2026-09-26 02:53:36 on dist-test-slave-ktng"
server_key: "58e78febced65dbd70e9531e7b3ebe4c"
server_key_iv: "d77edbfad9df210bda9b61705c869f8b"
server_key_version: "encryptionkey@0"
I20260926 02:53:36.867475 28052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "484aa4de22bf4443ad1cc14ec37366be"
format_stamp: "Formatted at 2026-09-26 02:53:36 on dist-test-slave-ktng"
server_key: "58e78febced65dbd70e9531e7b3ebe4c"
server_key_iv: "d77edbfad9df210bda9b61705c869f8b"
server_key_version: "encryptionkey@0"
I20260926 02:53:36.872648 28052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 02:53:36.876147 28067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:36.877434 28052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:53:36.877617 28052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "484aa4de22bf4443ad1cc14ec37366be"
format_stamp: "Formatted at 2026-09-26 02:53:36 on dist-test-slave-ktng"
server_key: "58e78febced65dbd70e9531e7b3ebe4c"
server_key_iv: "d77edbfad9df210bda9b61705c869f8b"
server_key_version: "encryptionkey@0"
I20260926 02:53:36.877831 28052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:36.894534 28052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:36.898829 28052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:36.899125 28052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:36.901743 28052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:36.901856 28052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:36.902017 28052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:36.902102 28052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:53:36 dist-test-slave-ktng krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391216, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
I20260926 02:53:36.927353 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:36.902542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:41587 (local address 127.19.185.254:37991)
0926 02:53:36.902719 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:36.902724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:36.904282 (+ 1558us) server_negotiation.cc:407] Connection header received
0926 02:53:36.907467 (+ 3185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:36.907475 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:36.907582 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:36.907739 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:36.910423 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.911469 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.913743 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.914342 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.914487 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:36.920332 (+ 5845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:36.920340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:36.920347 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:36.920394 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:36.923662 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:36.924580 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:36.924587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:36.924591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:36.924746 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:36.925355 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:36.925360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:36.925364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:36.925638 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:36.925758 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:36.926879 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0926 02:53:36.927056 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:53:36.929522 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:36.903120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:37991 (local address 127.19.185.194:41587)
0926 02:53:36.904084 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:36.904145 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:36.907014 (+ 2869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:36.908120 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:36.908136 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:36.909135 (+ 999us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:36.910211 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:36.910240 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.911618 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.911624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:36.913552 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:36.913563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.914634 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.914658 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:36.915012 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:36.916260 (+ 1248us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:36.916302 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:36.920092 (+ 3790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:36.923804 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:36.923814 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:36.923830 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:36.924456 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:36.924867 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:36.924873 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:36.924877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:36.925205 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:36.925774 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:36.925786 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:36.926369 (+ 583us) client_negotiation.cc:769] Sending connection context
0926 02:53:36.926774 (+ 405us) client_negotiation.cc:240] Negotiation successful
0926 02:53:36.927500 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":411,"threads_started":1}
I20260926 02:53:36.947254 28052 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:45653
I20260926 02:53:36.947274 28186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:45653 every 8 connection(s)
I20260926 02:53:36.948586 28052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:53:36.949189 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 28052
I20260926 02:53:36.949361 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:53:36.950325 20199 external_mini_cluster.cc:1428] Setting key 72cda5c1e4fc77975ac3793451149466
I20260926 02:53:36.952107 28187 heartbeater.cc:347] Connected to a master server at 127.19.185.254:37991
I20260926 02:53:36.952399 28187 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:36.953375 28187 heartbeater.cc:511] Master 127.19.185.254:37991 requested a full tablet report, sending...
I20260926 02:53:36.954802 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:36.938341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:55367 (local address 127.19.185.254:37991)
0926 02:53:36.938871 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:36.938875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:36.939783 (+ 908us) server_negotiation.cc:407] Connection header received
0926 02:53:36.940039 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:36.940046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:36.940145 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:36.940255 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:36.942018 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.943017 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.944457 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.945062 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.945215 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:36.947722 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:36.947727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:36.947730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:36.947759 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:36.951178 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:36.952599 (+ 1421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:36.952603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:36.952606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:36.952798 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:36.953199 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:36.953204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:36.953208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:36.953566 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:36.953797 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:36.954249 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 02:53:36.954452 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403}
I20260926 02:53:36.954869 28167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:36.938467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:37991 (local address 127.19.185.194:55367)
0926 02:53:36.939653 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:36.939694 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:36.939870 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:36.940305 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:36.940317 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:36.941179 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:36.941819 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:36.941836 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.943166 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.943175 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:36.944281 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:36.944293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:36.945283 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:36.945289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:36.945471 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:36.946187 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:36.946212 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:36.947545 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:36.951832 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:36.951838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:36.951842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:36.952431 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:36.952943 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:36.952948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:36.952952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:36.953079 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:36.953771 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:36.953777 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:36.953993 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 02:53:36.954155 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 02:53:36.954455 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"thread_start_us":760,"threads_started":1}
I20260926 02:53:36.954917 27841 ts_manager.cc:194] Registered new tserver with Master: 484aa4de22bf4443ad1cc14ec37366be (127.19.185.194:45653)
I20260926 02:53:36.956497 27841 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:41587
WARNING: no policy specified for oryx/127.19.185.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.195.keytab.
Entry for principal oryx/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.195.keytab.
I20260926 02:53:37.000409 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37991
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.195.keytab
--principal=oryx/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:41629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:37.325769 28191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:37.326027 28191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:37.326133 28191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:37.333228 28191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:37.333330 28191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:37.333411 28191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:53:37.333489 28191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:53:37.343861 28191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.195.keytab
--principal=oryx/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.185.254:37991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:37.345170 28191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:37.346473 28191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:37.360345 28196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:37.360394 28197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:37.360330 28199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:37.361325 28191 server_base.cc:1034] running on GCE node
I20260926 02:53:37.361965 28191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:37.362963 28191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:37.364230 28191 hybrid_clock.cc:648] HybridClock initialized: now 1790391217364129 us; error 79 us; skew 500 ppm
I20260926 02:53:37.364547 28191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:37 dist-test-slave-ktng krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391217, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:37.368014 28191 init.cc:377] Logged in from keytab as oryx/127.19.185.195@KRBTEST.COM (short username oryx)
I20260926 02:53:37.369639 28191 webserver.cc:460] Webserver started at http://127.19.185.195:39561/ using document root <none> and password file <none>
I20260926 02:53:37.370368 28191 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:37.370499 28191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:37.370837 28191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:37.373940 28191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "b057f1ce479845f5b29f1bd34f6f7284"
format_stamp: "Formatted at 2026-09-26 02:53:37 on dist-test-slave-ktng"
server_key: "9fd5c216de2c42b59d5fd4c84d5f0736"
server_key_iv: "00b459fe808e3177db569995aa4dad48"
server_key_version: "encryptionkey@0"
I20260926 02:53:37.374727 28191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "b057f1ce479845f5b29f1bd34f6f7284"
format_stamp: "Formatted at 2026-09-26 02:53:37 on dist-test-slave-ktng"
server_key: "9fd5c216de2c42b59d5fd4c84d5f0736"
server_key_iv: "00b459fe808e3177db569995aa4dad48"
server_key_version: "encryptionkey@0"
I20260926 02:53:37.379954 28191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:53:37.383885 28206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:37.385353 28191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:53:37.385555 28191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "b057f1ce479845f5b29f1bd34f6f7284"
format_stamp: "Formatted at 2026-09-26 02:53:37 on dist-test-slave-ktng"
server_key: "9fd5c216de2c42b59d5fd4c84d5f0736"
server_key_iv: "00b459fe808e3177db569995aa4dad48"
server_key_version: "encryptionkey@0"
I20260926 02:53:37.385798 28191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:37.420993 28191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:37.425418 28191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:37.425776 28191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:37.429579 28191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:37.429755 28191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 02:53:37.429941 28191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:37.430045 28191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:37.440759 28048 heartbeater.cc:503] Master 127.19.185.254:37991 was elected leader, sending a full tablet report...
Sep 26 02:53:37 dist-test-slave-ktng krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391217, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
I20260926 02:53:37.454072 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.429603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:57209 (local address 127.19.185.254:37991)
0926 02:53:37.429835 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:37.429842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:37.431660 (+ 1818us) server_negotiation.cc:407] Connection header received
0926 02:53:37.434200 (+ 2540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:37.434206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:37.434302 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:37.434436 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:37.436595 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.437968 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.440649 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.441262 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.441390 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.446372 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:37.446383 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:37.446386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:37.446419 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:37.450228 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:37.451146 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:37.451178 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:37.451185 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:37.451412 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:37.452046 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:37.452053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:37.452059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:37.452420 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:37.452579 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:37.453431 (+ 852us) server_negotiation.cc:299] Negotiation successful
0926 02:53:37.453692 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:53:37.455170 28215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.430387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:37991 (local address 127.19.185.195:57209)
0926 02:53:37.431488 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:37.431537 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:37.433819 (+ 2282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:37.434643 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:37.434652 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:37.435361 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:37.436288 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.436312 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.438207 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.438217 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.440415 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.440427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.441495 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.441502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.441752 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.442611 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:37.442672 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:37.446121 (+ 3449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:37.450527 (+ 4406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:37.450534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:37.450548 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:37.451001 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:37.451556 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:37.451563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:37.451571 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:37.451874 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:37.452556 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:37.452564 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:37.452961 (+ 397us) client_negotiation.cc:769] Sending connection context
0926 02:53:37.453309 (+ 348us) client_negotiation.cc:240] Negotiation successful
0926 02:53:37.453868 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":2510,"threads_started":1}
I20260926 02:53:37.477540 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.462055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:36427 (local address 127.19.185.254:37991)
0926 02:53:37.462289 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:37.462298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:37.463921 (+ 1623us) server_negotiation.cc:407] Connection header received
0926 02:53:37.464147 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:37.464153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:37.464224 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:37.464336 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:37.465325 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.466094 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.467577 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.468336 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.468509 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.471511 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:37.471520 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:37.471527 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:37.471567 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:37.474350 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:37.475406 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:37.475411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:37.475413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:37.475561 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:37.476099 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:37.476103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:37.476105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:37.476384 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:37.476509 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:37.477090 (+ 581us) server_negotiation.cc:299] Negotiation successful
0926 02:53:37.477290 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:53:37.478161 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.462028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:37991 (local address 127.19.185.195:36427)
0926 02:53:37.463800 (+ 1772us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:37.463825 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:37.464007 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:37.464359 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:37.464365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:37.464865 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:37.465172 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.465181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.466239 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.466245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.467414 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.467426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.468788 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.468793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.468963 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.469748 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:37.469780 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:37.471242 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:37.474483 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:37.474492 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:37.474499 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:37.475181 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:37.475684 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:37.475692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:37.475698 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:37.475886 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:37.476503 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:37.476511 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:37.476777 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 02:53:37.476960 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 02:53:37.477407 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1617,"thread_start_us":1475,"threads_started":1}
I20260926 02:53:37.493072 28191 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:45975
I20260926 02:53:37.493100 28325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:45975 every 8 connection(s)
I20260926 02:53:37.494423 28191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:53:37.496572 28326 heartbeater.cc:347] Connected to a master server at 127.19.185.254:37991
I20260926 02:53:37.496766 28326 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:37.497359 28326 heartbeater.cc:511] Master 127.19.185.254:37991 requested a full tablet report, sending...
I20260926 02:53:37.498615 27841 ts_manager.cc:194] Registered new tserver with Master: b057f1ce479845f5b29f1bd34f6f7284 (127.19.185.195:45975)
I20260926 02:53:37.499497 27841 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:57209
I20260926 02:53:37.502362 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 28191
I20260926 02:53:37.502516 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:53:37.502900 20199 external_mini_cluster.cc:1428] Setting key b5ffe83cf406689fb775fee267752d1c
I20260926 02:53:37.505770 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:53:37 dist-test-slave-ktng krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391217, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:53:37 dist-test-slave-ktng krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:37 dist-test-slave-ktng krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:53:37.555697 27935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:37.544603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34286 (local address 127.19.185.254:37991)
0926 02:53:37.544841 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:37.544848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:37.545061 (+ 213us) server_negotiation.cc:407] Connection header received
0926 02:53:37.545348 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:37.545353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:37.545418 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:37.545527 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:37.546843 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.547654 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.549071 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.549558 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.549725 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.555454 (+ 5729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34286: BlockingRecv error: recv got EOF from 127.0.0.1:34286 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 02:53:37 dist-test-slave-ktng krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391217, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
I20260926 02:53:37.571502 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.558771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.19.185.254:37991)
0926 02:53:37.558933 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:37.558938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:37.559342 (+ 404us) server_negotiation.cc:407] Connection header received
0926 02:53:37.559507 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:37.559511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:37.559585 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:37.559670 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:37.560709 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.561410 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.562561 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.562959 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.563049 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.566662 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:37.566667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:37.566671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:37.566702 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:37.569190 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:37.569797 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:37.569801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:37.569803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:37.569920 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:37.570311 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:37.570314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:37.570316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:37.570579 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:37.570795 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:37.570999 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 02:53:37.571178 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:53:37.575743 27841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:53:37.582625 27841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:53:37.615583 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.594453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:45975 (local address 127.0.0.1:49636)
0926 02:53:37.606305 (+ 11852us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:37.606369 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:37.606617 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:37.608527 (+ 1910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:37.608535 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:37.608576 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:37.609120 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.609132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.611000 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.611008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.613198 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.613210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.614579 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.614586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.614757 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.614806 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 02:53:37.614938 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 02:53:37.615115 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11502,"mutex_wait_us":21,"spinlock_wait_cycles":768,"thread_start_us":13061,"threads_started":1}
I20260926 02:53:37.616640 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.594953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.19.185.195:45975)
0926 02:53:37.605827 (+ 10874us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:37.605840 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:37.607770 (+ 1930us) server_negotiation.cc:407] Connection header received
0926 02:53:37.607888 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:37.607898 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:37.608245 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:37.609251 (+ 1006us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:37.609631 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.610866 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.613358 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.614384 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.614573 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.614758 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:37.616127 (+ 1369us) server_negotiation.cc:299] Negotiation successful
0926 02:53:37.616293 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10606,"thread_start_us":10254,"threads_started":1}
I20260926 02:53:37.616667 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.594647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:45653 (local address 127.0.0.1:56924)
0926 02:53:37.607802 (+ 13155us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:37.607820 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:37.607937 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:37.608569 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:37.608574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:37.608599 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:37.609707 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.609717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.611582 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.611587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.614128 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.614142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.615857 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.615862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.616026 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.616066 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 02:53:37.616130 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 02:53:37.616253 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13006,"mutex_wait_us":77,"thread_start_us":11238,"threads_started":1}
I20260926 02:53:37.617093 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.607815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:39795 (local address 127.0.0.1:59446)
0926 02:53:37.608472 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:37.608488 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:37.608589 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:37.609244 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:37.609248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:37.609278 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:37.609794 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.609808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.611634 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.611645 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.614128 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.614142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.616295 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.616301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.616424 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.616445 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 02:53:37.616561 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 02:53:37.616709 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":136,"spinlock_wait_cycles":7424,"thread_start_us":345,"threads_started":1}
I20260926 02:53:37.617206 28348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.607227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.19.185.193:39795)
0926 02:53:37.607768 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:37.607778 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:37.608723 (+ 945us) server_negotiation.cc:407] Connection header received
0926 02:53:37.608824 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:37.608832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:37.609062 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:37.609242 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:37.609984 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.611466 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.614295 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.615783 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.615948 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.616144 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:37.616604 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 02:53:37.616896 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":282,"threads_started":1}
I20260926 02:53:37.618494 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.594839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.19.185.194:45653)
0926 02:53:37.607325 (+ 12486us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:37.607338 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:37.607930 (+ 592us) server_negotiation.cc:407] Connection header received
0926 02:53:37.608102 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:37.608113 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:37.608357 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:37.608775 (+ 418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:37.609873 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.611416 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.614306 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.615627 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.616923 (+ 1296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.617959 (+ 1036us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:37.618029 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 02:53:37.618186 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12268,"thread_start_us":12091,"threads_started":1}
I20260926 02:53:37.620842 27978 tablet_service.cc:1467] Processing CreateTablet for tablet 01bd9711dac94dc385c2a36069a0bf09 (DEFAULT_TABLE table=test-table [id=a44bab2a73124f45add69bf04b707b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:37.620873 28256 tablet_service.cc:1467] Processing CreateTablet for tablet 01bd9711dac94dc385c2a36069a0bf09 (DEFAULT_TABLE table=test-table [id=a44bab2a73124f45add69bf04b707b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:37.621528 28117 tablet_service.cc:1467] Processing CreateTablet for tablet 01bd9711dac94dc385c2a36069a0bf09 (DEFAULT_TABLE table=test-table [id=a44bab2a73124f45add69bf04b707b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:37.622078 27978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01bd9711dac94dc385c2a36069a0bf09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:37.622257 28256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01bd9711dac94dc385c2a36069a0bf09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:37.622731 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01bd9711dac94dc385c2a36069a0bf09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:37.633029 28350 tablet_bootstrap.cc:492] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3: Bootstrap starting.
I20260926 02:53:37.636247 28352 tablet_bootstrap.cc:492] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284: Bootstrap starting.
I20260926 02:53:37.636821 28350 tablet_bootstrap.cc:654] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:37.636988 28351 tablet_bootstrap.cc:492] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be: Bootstrap starting.
I20260926 02:53:37.638273 28350 log.cc:824] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:37.640333 28351 tablet_bootstrap.cc:654] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:37.641069 28350 tablet_bootstrap.cc:492] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3: No bootstrap required, opened a new log
I20260926 02:53:37.641417 28350 ts_tablet_manager.cc:1397] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 02:53:37.642016 28351 log.cc:824] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:37.642279 28352 tablet_bootstrap.cc:654] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:37.643479 28352 log.cc:824] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:37.645120 28351 tablet_bootstrap.cc:492] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be: No bootstrap required, opened a new log
I20260926 02:53:37.645380 28351 ts_tablet_manager.cc:1397] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:53:37.646339 28352 tablet_bootstrap.cc:492] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284: No bootstrap required, opened a new log
I20260926 02:53:37.646575 28352 ts_tablet_manager.cc:1397] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260926 02:53:37.649899 28350 raft_consensus.cc:348] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } }
I20260926 02:53:37.650465 28350 raft_consensus.cc:374] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:37.650586 28350 raft_consensus.cc:729] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05960a65b34b4cd7ad9f4a61855240d3, State: Initialized, Role: FOLLOWER
I20260926 02:53:37.651235 28350 consensus_queue.cc:260] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } }
I20260926 02:53:37.654054 28350 ts_tablet_manager.cc:1428] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260926 02:53:37.656967 28352 raft_consensus.cc:348] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } }
I20260926 02:53:37.657469 28352 raft_consensus.cc:374] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:37.657601 28352 raft_consensus.cc:729] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b057f1ce479845f5b29f1bd34f6f7284, State: Initialized, Role: FOLLOWER
I20260926 02:53:37.658241 28352 consensus_queue.cc:260] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } }
I20260926 02:53:37.658453 28351 raft_consensus.cc:348] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } }
I20260926 02:53:37.658838 28351 raft_consensus.cc:374] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:37.658957 28351 raft_consensus.cc:729] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484aa4de22bf4443ad1cc14ec37366be, State: Initialized, Role: FOLLOWER
I20260926 02:53:37.659322 28326 heartbeater.cc:503] Master 127.19.185.254:37991 was elected leader, sending a full tablet report...
I20260926 02:53:37.659547 28351 consensus_queue.cc:260] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } }
I20260926 02:53:37.659837 28352 ts_tablet_manager.cc:1428] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260926 02:53:37.660620 28187 heartbeater.cc:503] Master 127.19.185.254:37991 was elected leader, sending a full tablet report...
I20260926 02:53:37.661228 28351 ts_tablet_manager.cc:1428] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260926 02:53:37.674804 28356 raft_consensus.cc:482] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:53:37.675104 28356 raft_consensus.cc:504] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } }
I20260926 02:53:37.676757 28356 leader_election.cc:288] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b057f1ce479845f5b29f1bd34f6f7284 (127.19.185.195:45975), 484aa4de22bf4443ad1cc14ec37366be (127.19.185.194:45653)
W20260926 02:53:37.681216 28049 tablet.cc:2369] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:53:37.685839 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.677232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:36621 (local address 127.19.185.194:45653)
0926 02:53:37.677419 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:37.677425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:37.678125 (+ 700us) server_negotiation.cc:407] Connection header received
0926 02:53:37.678291 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:37.678296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:37.678368 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:37.678462 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:37.679245 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.680339 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.683399 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.684655 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.684802 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.684875 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:37.685447 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 02:53:37.685582 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:53:37.685839 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.677243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:51213 (local address 127.19.185.195:45975)
0926 02:53:37.677419 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:37.677425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:37.678125 (+ 700us) server_negotiation.cc:407] Connection header received
0926 02:53:37.678291 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:37.678296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:37.678368 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:37.678462 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:37.679246 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.680390 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.683393 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.684613 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.684802 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.684875 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:37.685451 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 02:53:37.685582 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:53:37.686060 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.677285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:45975 (local address 127.19.185.193:51213)
0926 02:53:37.678017 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:37.678034 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:37.678154 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:37.678496 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:37.678501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:37.678546 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:37.679071 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.679090 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.680495 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.680499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.683201 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.683216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.684894 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.684901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.685175 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.685267 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 02:53:37.685368 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 02:53:37.685577 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":350,"spinlock_wait_cycles":5504,"thread_start_us":384,"threads_started":1}
I20260926 02:53:37.686180 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.677007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:45653 (local address 127.19.185.193:36621)
0926 02:53:37.677980 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:37.678006 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:37.678161 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:37.678496 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:37.678501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:37.678546 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:37.679098 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.679107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.680490 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.680495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.683201 (+ 2706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.683216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.684909 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.684914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.685170 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.685269 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 02:53:37.685368 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 02:53:37.685576 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":216,"spinlock_wait_cycles":14720,"thread_start_us":624,"threads_started":1}
I20260926 02:53:37.687517 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01bd9711dac94dc385c2a36069a0bf09" candidate_uuid: "05960a65b34b4cd7ad9f4a61855240d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484aa4de22bf4443ad1cc14ec37366be" is_pre_election: true
I20260926 02:53:37.687517 28281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01bd9711dac94dc385c2a36069a0bf09" candidate_uuid: "05960a65b34b4cd7ad9f4a61855240d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b057f1ce479845f5b29f1bd34f6f7284" is_pre_election: true
I20260926 02:53:37.688079 28137 raft_consensus.cc:2454] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05960a65b34b4cd7ad9f4a61855240d3 in term 0.
I20260926 02:53:37.688079 28281 raft_consensus.cc:2454] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05960a65b34b4cd7ad9f4a61855240d3 in term 0.
I20260926 02:53:37.688825 27930 leader_election.cc:302] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05960a65b34b4cd7ad9f4a61855240d3, 484aa4de22bf4443ad1cc14ec37366be, b057f1ce479845f5b29f1bd34f6f7284; no voters:
I20260926 02:53:37.689301 28356 raft_consensus.cc:2781] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:53:37.689457 28356 raft_consensus.cc:482] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:53:37.689579 28356 raft_consensus.cc:3037] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:37.692126 28356 raft_consensus.cc:504] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } }
I20260926 02:53:37.692677 28356 leader_election.cc:288] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [CANDIDATE]: Term 1 election: Requested vote from peers b057f1ce479845f5b29f1bd34f6f7284 (127.19.185.195:45975), 484aa4de22bf4443ad1cc14ec37366be (127.19.185.194:45653)
I20260926 02:53:37.693181 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01bd9711dac94dc385c2a36069a0bf09" candidate_uuid: "05960a65b34b4cd7ad9f4a61855240d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484aa4de22bf4443ad1cc14ec37366be"
I20260926 02:53:37.693192 28281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01bd9711dac94dc385c2a36069a0bf09" candidate_uuid: "05960a65b34b4cd7ad9f4a61855240d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b057f1ce479845f5b29f1bd34f6f7284"
I20260926 02:53:37.693392 28137 raft_consensus.cc:3037] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:37.693392 28281 raft_consensus.cc:3037] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:37.696671 28137 raft_consensus.cc:2454] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05960a65b34b4cd7ad9f4a61855240d3 in term 1.
I20260926 02:53:37.696671 28281 raft_consensus.cc:2454] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05960a65b34b4cd7ad9f4a61855240d3 in term 1.
I20260926 02:53:37.697141 27929 leader_election.cc:302] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05960a65b34b4cd7ad9f4a61855240d3, 484aa4de22bf4443ad1cc14ec37366be, b057f1ce479845f5b29f1bd34f6f7284; no voters:
I20260926 02:53:37.697542 28356 raft_consensus.cc:2781] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:37.698001 28356 raft_consensus.cc:686] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 LEADER]: Becoming Leader. State: Replica: 05960a65b34b4cd7ad9f4a61855240d3, State: Running, Role: LEADER
I20260926 02:53:37.698449 28356 consensus_queue.cc:237] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } }
W20260926 02:53:37.702225 28188 tablet.cc:2369] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:53:37.702798 27841 catalog_manager.cc:5495] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05960a65b34b4cd7ad9f4a61855240d3 (127.19.185.193). New cstate: current_term: 1 leader_uuid: "05960a65b34b4cd7ad9f4a61855240d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } health_report { overall_health: HEALTHY } } }
I20260926 02:53:37.721791 28348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.715948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.19.185.193:39795)
0926 02:53:37.716156 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:37.716163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:37.716262 (+ 99us) server_negotiation.cc:407] Connection header received
0926 02:53:37.716505 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:37.716510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:37.716599 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:37.716777 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:53:37.717361 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.718431 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.719816 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.720188 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.720281 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:37.720630 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:53:37.721160 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:53:37.721265 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:37.721419 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 02:53:37.721483 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 02:53:37.729450 28137 raft_consensus.cc:1261] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Refusing update from remote peer 05960a65b34b4cd7ad9f4a61855240d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:37.729473 28281 raft_consensus.cc:1261] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Refusing update from remote peer 05960a65b34b4cd7ad9f4a61855240d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:37.730417 28361 consensus_queue.cc:1035] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:37.730772 28356 consensus_queue.cc:1035] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:37.741340 28364 mvcc.cc:205] Tried to move back new op lower bound from 7333442427810525184 to 7333442427696594944. Current Snapshot: MvccSnapshot[applied={T|T < 7333442427810525184}]
I20260926 02:53:37.753970 27841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:37.754258 27841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:37.758178 27841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f: Sending DeleteTablet for 3 replicas of tablet 01bd9711dac94dc385c2a36069a0bf09
I20260926 02:53:37.759281 28256 tablet_service.cc:1514] Processing DeleteTablet for tablet 01bd9711dac94dc385c2a36069a0bf09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:37 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:49636
I20260926 02:53:37.759505 28117 tablet_service.cc:1514] Processing DeleteTablet for tablet 01bd9711dac94dc385c2a36069a0bf09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:37 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:56924
I20260926 02:53:37.760310 27841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 02:53:37.760443 27978 tablet_service.cc:1514] Processing DeleteTablet for tablet 01bd9711dac94dc385c2a36069a0bf09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:37 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:59446
W20260926 02:53:37.761056 27841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:53:37.768462 28255 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfb5018d5274ba58cc72a11e713a9b1 (DEFAULT_TABLE table=test-table [id=7549f588bc0e43f498e5ba3c7ddddb5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:37.769090 28255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfb5018d5274ba58cc72a11e713a9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:37.769124 28116 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfb5018d5274ba58cc72a11e713a9b1 (DEFAULT_TABLE table=test-table [id=7549f588bc0e43f498e5ba3c7ddddb5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:37.769789 28116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfb5018d5274ba58cc72a11e713a9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:37.770583 28373 tablet_replica.cc:330] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284: stopping tablet replica
I20260926 02:53:37.771030 28373 raft_consensus.cc:2229] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:37.772450 28373 raft_consensus.cc:2258] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:37.773978 27977 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfb5018d5274ba58cc72a11e713a9b1 (DEFAULT_TABLE table=test-table [id=7549f588bc0e43f498e5ba3c7ddddb5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:37.774571 27977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfb5018d5274ba58cc72a11e713a9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:37.775691 28373 ts_tablet_manager.cc:1905] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:37.777424 28351 tablet_bootstrap.cc:492] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be: Bootstrap starting.
I20260926 02:53:37.779563 28351 tablet_bootstrap.cc:654] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:37.784337 28373 ts_tablet_manager.cc:1918] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:37.784538 28350 tablet_bootstrap.cc:492] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3: Bootstrap starting.
I20260926 02:53:37.785511 28373 log.cc:1196] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/01bd9711dac94dc385c2a36069a0bf09
I20260926 02:53:37.786041 28373 ts_tablet_manager.cc:1939] T 01bd9711dac94dc385c2a36069a0bf09 P b057f1ce479845f5b29f1bd34f6f7284: Deleting consensus metadata
I20260926 02:53:37.786160 28350 tablet_bootstrap.cc:654] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:37.788089 27827 catalog_manager.cc:4841] TS b057f1ce479845f5b29f1bd34f6f7284 (127.19.185.195:45975): tablet 01bd9711dac94dc385c2a36069a0bf09 (table test-table [id=a44bab2a73124f45add69bf04b707b92]) successfully deleted
I20260926 02:53:37.788986 28375 tablet_replica.cc:330] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3: stopping tablet replica
I20260926 02:53:37.789634 28352 tablet_bootstrap.cc:492] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284: Bootstrap starting.
I20260926 02:53:37.789716 28375 raft_consensus.cc:2229] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:53:37.790983 28374 tablet_replica.cc:330] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be: stopping tablet replica
I20260926 02:53:37.791060 28352 tablet_bootstrap.cc:654] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:37.791527 28374 raft_consensus.cc:2229] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:37.792129 28374 raft_consensus.cc:2258] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:37.792730 28375 raft_consensus.cc:2258] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:37.793066 28352 tablet_bootstrap.cc:492] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284: No bootstrap required, opened a new log
I20260926 02:53:37.793216 28352 ts_tablet_manager.cc:1397] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 02:53:37.794097 28352 raft_consensus.cc:348] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:37.794302 28352 raft_consensus.cc:374] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:37.794391 28352 raft_consensus.cc:729] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b057f1ce479845f5b29f1bd34f6f7284, State: Initialized, Role: FOLLOWER
I20260926 02:53:37.794687 28352 consensus_queue.cc:260] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:37.794920 28374 ts_tablet_manager.cc:1905] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:37.794920 28375 ts_tablet_manager.cc:1905] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:37.797051 28352 ts_tablet_manager.cc:1428] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260926 02:53:37.801831 28375 ts_tablet_manager.cc:1918] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:37.801982 28375 log.cc:1196] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/01bd9711dac94dc385c2a36069a0bf09
I20260926 02:53:37.805960 28350 tablet_bootstrap.cc:492] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3: No bootstrap required, opened a new log
I20260926 02:53:37.806384 28350 ts_tablet_manager.cc:1397] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.001s
I20260926 02:53:37.807281 28350 raft_consensus.cc:348] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:37.808261 28351 tablet_bootstrap.cc:492] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be: No bootstrap required, opened a new log
I20260926 02:53:37.808406 28351 ts_tablet_manager.cc:1397] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.018s
I20260926 02:53:37.809243 28351 raft_consensus.cc:348] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:37.809511 28351 raft_consensus.cc:374] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:37.809669 28351 raft_consensus.cc:729] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484aa4de22bf4443ad1cc14ec37366be, State: Initialized, Role: FOLLOWER
I20260926 02:53:37.810077 28351 consensus_queue.cc:260] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:37.813249 28351 ts_tablet_manager.cc:1428] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20260926 02:53:37.813278 28374 ts_tablet_manager.cc:1918] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:37.813519 28374 log.cc:1196] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/01bd9711dac94dc385c2a36069a0bf09
I20260926 02:53:37.814158 28374 ts_tablet_manager.cc:1939] T 01bd9711dac94dc385c2a36069a0bf09 P 484aa4de22bf4443ad1cc14ec37366be: Deleting consensus metadata
I20260926 02:53:37.815399 28350 raft_consensus.cc:374] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:37.815522 28350 raft_consensus.cc:729] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05960a65b34b4cd7ad9f4a61855240d3, State: Initialized, Role: FOLLOWER
I20260926 02:53:37.815807 28350 consensus_queue.cc:260] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:37.816094 27826 catalog_manager.cc:4841] TS 484aa4de22bf4443ad1cc14ec37366be (127.19.185.194:45653): tablet 01bd9711dac94dc385c2a36069a0bf09 (table test-table [id=a44bab2a73124f45add69bf04b707b92]) successfully deleted
I20260926 02:53:37.816350 28350 ts_tablet_manager.cc:1428] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20260926 02:53:37.817056 28375 ts_tablet_manager.cc:1939] T 01bd9711dac94dc385c2a36069a0bf09 P 05960a65b34b4cd7ad9f4a61855240d3: Deleting consensus metadata
I20260926 02:53:37.818661 27826 catalog_manager.cc:4841] TS 05960a65b34b4cd7ad9f4a61855240d3 (127.19.185.193:39795): tablet 01bd9711dac94dc385c2a36069a0bf09 (table test-table [id=a44bab2a73124f45add69bf04b707b92]) successfully deleted
I20260926 02:53:37.993031 28358 raft_consensus.cc:482] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:53:37.993319 28358 raft_consensus.cc:504] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:37.994956 28358 leader_election.cc:288] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05960a65b34b4cd7ad9f4a61855240d3 (127.19.185.193:39795), b057f1ce479845f5b29f1bd34f6f7284 (127.19.185.195:45975)
W20260926 02:53:37.997079 28327 tablet.cc:2369] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:53:38.001441 28348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.995685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:60157 (local address 127.19.185.193:39795)
0926 02:53:37.995860 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:37.995866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:37.995891 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:53:37.996065 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:37.996070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:37.996145 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:37.996201 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:37.996838 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.997814 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.999585 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.000400 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.000590 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.000678 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:38.000964 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 02:53:38.001111 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:53:38.001439 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.995148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:39795 (local address 127.19.185.194:60157)
0926 02:53:37.995725 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:37.995747 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:37.995889 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:37.996358 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:37.996361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:37.996382 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:37.996665 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.996674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:37.997959 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:37.997967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:37.999428 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:37.999441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.000626 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.000633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.000800 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.000823 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:53:38.000929 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 02:53:38.001062 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":746,"threads_started":1}
I20260926 02:53:38.002787 27998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfb5018d5274ba58cc72a11e713a9b1" candidate_uuid: "484aa4de22bf4443ad1cc14ec37366be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05960a65b34b4cd7ad9f4a61855240d3" is_pre_election: true
I20260926 02:53:38.003264 27998 raft_consensus.cc:2454] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484aa4de22bf4443ad1cc14ec37366be in term 0.
I20260926 02:53:38.003968 28069 leader_election.cc:302] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05960a65b34b4cd7ad9f4a61855240d3, 484aa4de22bf4443ad1cc14ec37366be; no voters:
I20260926 02:53:38.004438 28358 raft_consensus.cc:2781] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:53:38.004581 28358 raft_consensus.cc:482] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:53:38.004683 28358 raft_consensus.cc:3037] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:38.007412 28358 raft_consensus.cc:504] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.008154 28358 leader_election.cc:288] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [CANDIDATE]: Term 1 election: Requested vote from peers 05960a65b34b4cd7ad9f4a61855240d3 (127.19.185.193:39795), b057f1ce479845f5b29f1bd34f6f7284 (127.19.185.195:45975)
I20260926 02:53:38.008824 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.995679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:35435 (local address 127.19.185.195:45975)
0926 02:53:37.995861 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:37.995866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:38.001778 (+ 5912us) server_negotiation.cc:407] Connection header received
0926 02:53:38.002033 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:38.002039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:38.002135 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:38.002204 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:38.002804 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.003881 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.006352 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.007584 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.007748 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.007815 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:38.008429 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 02:53:38.008564 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:53:38.008848 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:37.995314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:45975 (local address 127.19.185.194:35435)
0926 02:53:38.001693 (+ 6379us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:38.001712 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:38.001872 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:38.002399 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:38.002404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:38.002428 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:38.002616 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:38.002628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.004029 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.004034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.006162 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:38.006179 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.007907 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.007913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.008190 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.008265 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 02:53:38.008401 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 02:53:38.008541 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6273,"thread_start_us":7295,"threads_started":1}
I20260926 02:53:38.008813 27998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfb5018d5274ba58cc72a11e713a9b1" candidate_uuid: "484aa4de22bf4443ad1cc14ec37366be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05960a65b34b4cd7ad9f4a61855240d3"
I20260926 02:53:38.009029 27998 raft_consensus.cc:3037] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:38.011778 28281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfb5018d5274ba58cc72a11e713a9b1" candidate_uuid: "484aa4de22bf4443ad1cc14ec37366be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b057f1ce479845f5b29f1bd34f6f7284" is_pre_election: true
I20260926 02:53:38.011983 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfb5018d5274ba58cc72a11e713a9b1" candidate_uuid: "484aa4de22bf4443ad1cc14ec37366be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b057f1ce479845f5b29f1bd34f6f7284"
I20260926 02:53:38.012055 28281 raft_consensus.cc:2454] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484aa4de22bf4443ad1cc14ec37366be in term 0.
I20260926 02:53:38.013262 27998 raft_consensus.cc:2454] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484aa4de22bf4443ad1cc14ec37366be in term 1.
I20260926 02:53:38.013691 28069 leader_election.cc:302] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05960a65b34b4cd7ad9f4a61855240d3, 484aa4de22bf4443ad1cc14ec37366be; no voters: b057f1ce479845f5b29f1bd34f6f7284
I20260926 02:53:38.014034 28358 raft_consensus.cc:2781] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:38.014559 28358 raft_consensus.cc:686] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [term 1 LEADER]: Becoming Leader. State: Replica: 484aa4de22bf4443ad1cc14ec37366be, State: Running, Role: LEADER
I20260926 02:53:38.014995 28358 consensus_queue.cc:237] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.018711 27840 catalog_manager.cc:5495] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be reported cstate change: term changed from 0 to 1, leader changed from <none> to 484aa4de22bf4443ad1cc14ec37366be (127.19.185.194). New cstate: current_term: 1 leader_uuid: "484aa4de22bf4443ad1cc14ec37366be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } health_report { overall_health: UNKNOWN } } }
Sep 26 02:53:38 dist-test-slave-ktng krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391215, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.254@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
I20260926 02:53:38.120612 28388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.097859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.19.185.254:37991)
0926 02:53:38.099051 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:38.099055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:38.099074 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:53:38.099118 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:38.099123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:38.099173 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:38.099261 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:38.100990 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.101790 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.103736 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.104122 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.104235 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.114524 (+ 10289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:38.114529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:38.114533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:38.114564 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:38.117363 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:38.118359 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:38.118366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:38.118370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:38.118534 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:38.118969 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:38.118972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:38.118974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:38.119277 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:38.119452 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:38.120084 (+ 632us) server_negotiation.cc:299] Negotiation successful
0926 02:53:38.120351 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":956,"threads_started":1}
I20260926 02:53:38.120699 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.097859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:37991 (local address 127.0.0.1:34300)
0926 02:53:38.098642 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:38.098666 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:38.098829 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:38.099381 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:38.099386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:38.100416 (+ 1030us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:38.100798 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:38.100812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.101938 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.101944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.103571 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:38.103586 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.104339 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.104347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.104519 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.111218 (+ 6699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:38.111246 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:38.114276 (+ 3030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:38.117563 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:38.117576 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:38.117585 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:38.118202 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:38.118663 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:38.118668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:38.118672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:38.118849 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:38.119470 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:38.119480 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:38.119717 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 02:53:38.119917 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 02:53:38.120218 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"spinlock_wait_cycles":17280,"thread_start_us":406,"threads_started":1}
I20260926 02:53:38.126219 27840 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:34300:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 02:53:38.127532 27840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:53:38.134383 27978 tablet_service.cc:1467] Processing CreateTablet for tablet 233ced9a2d4b424cab10252f695327b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=652fd48ea6d74239892645f28cac4f66]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:53:38.134505 28117 tablet_service.cc:1467] Processing CreateTablet for tablet 233ced9a2d4b424cab10252f695327b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=652fd48ea6d74239892645f28cac4f66]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:53:38.135013 27978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233ced9a2d4b424cab10252f695327b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:38.135076 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233ced9a2d4b424cab10252f695327b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:38.135787 28255 tablet_service.cc:1467] Processing CreateTablet for tablet 233ced9a2d4b424cab10252f695327b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=652fd48ea6d74239892645f28cac4f66]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:53:38.136252 28255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233ced9a2d4b424cab10252f695327b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:38.142565 28351 tablet_bootstrap.cc:492] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be: Bootstrap starting.
I20260926 02:53:38.142843 28350 tablet_bootstrap.cc:492] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3: Bootstrap starting.
I20260926 02:53:38.143275 28352 tablet_bootstrap.cc:492] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284: Bootstrap starting.
I20260926 02:53:38.144380 28351 tablet_bootstrap.cc:654] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:38.144681 28350 tablet_bootstrap.cc:654] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:38.144996 28352 tablet_bootstrap.cc:654] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:38.154060 28351 tablet_bootstrap.cc:492] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be: No bootstrap required, opened a new log
I20260926 02:53:38.154340 28351 ts_tablet_manager.cc:1397] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20260926 02:53:38.155427 28350 tablet_bootstrap.cc:492] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3: No bootstrap required, opened a new log
I20260926 02:53:38.155664 28350 ts_tablet_manager.cc:1397] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260926 02:53:38.155997 28351 raft_consensus.cc:348] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.156394 28351 raft_consensus.cc:374] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:38.156522 28351 raft_consensus.cc:729] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484aa4de22bf4443ad1cc14ec37366be, State: Initialized, Role: FOLLOWER
I20260926 02:53:38.156493 28350 raft_consensus.cc:348] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.156831 28350 raft_consensus.cc:374] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:38.157096 28350 raft_consensus.cc:729] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05960a65b34b4cd7ad9f4a61855240d3, State: Initialized, Role: FOLLOWER
I20260926 02:53:38.157251 28352 tablet_bootstrap.cc:492] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284: No bootstrap required, opened a new log
I20260926 02:53:38.157063 28351 consensus_queue.cc:260] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.157413 28352 ts_tablet_manager.cc:1397] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20260926 02:53:38.157411 28350 consensus_queue.cc:260] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.157730 28358 tablet_replica.cc:439] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } }
I20260926 02:53:38.158048 28358 tablet_replica.cc:442] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:53:38.157827 28361 tablet_replica.cc:439] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } }
I20260926 02:53:38.158187 28352 raft_consensus.cc:348] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.158402 28352 raft_consensus.cc:374] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:38.158473 28352 raft_consensus.cc:729] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b057f1ce479845f5b29f1bd34f6f7284, State: Initialized, Role: FOLLOWER
I20260926 02:53:38.158704 28350 ts_tablet_manager.cc:1428] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 02:53:38.158747 28352 consensus_queue.cc:260] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.158896 28361 tablet_replica.cc:442] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:53:38.159171 28357 tablet_replica.cc:439] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } }
I20260926 02:53:38.159472 28357 tablet_replica.cc:442] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:53:38.160265 28351 ts_tablet_manager.cc:1428] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260926 02:53:38.161828 28352 ts_tablet_manager.cc:1428] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260926 02:53:38.223959 28361 raft_consensus.cc:482] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:53:38.224179 28361 raft_consensus.cc:504] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.224799 28361 leader_election.cc:288] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 484aa4de22bf4443ad1cc14ec37366be (127.19.185.194:45653), b057f1ce479845f5b29f1bd34f6f7284 (127.19.185.195:45975)
I20260926 02:53:38.225400 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ced9a2d4b424cab10252f695327b0" candidate_uuid: "05960a65b34b4cd7ad9f4a61855240d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b057f1ce479845f5b29f1bd34f6f7284" is_pre_election: true
I20260926 02:53:38.225548 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ced9a2d4b424cab10252f695327b0" candidate_uuid: "05960a65b34b4cd7ad9f4a61855240d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484aa4de22bf4443ad1cc14ec37366be" is_pre_election: true
I20260926 02:53:38.225659 28280 raft_consensus.cc:2454] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05960a65b34b4cd7ad9f4a61855240d3 in term 0.
I20260926 02:53:38.225835 28137 raft_consensus.cc:2454] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05960a65b34b4cd7ad9f4a61855240d3 in term 0.
I20260926 02:53:38.226091 27930 leader_election.cc:302] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05960a65b34b4cd7ad9f4a61855240d3, b057f1ce479845f5b29f1bd34f6f7284; no voters:
I20260926 02:53:38.226392 28361 raft_consensus.cc:2781] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:53:38.226472 28361 raft_consensus.cc:482] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:53:38.226630 28361 raft_consensus.cc:3037] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:38.228884 28361 raft_consensus.cc:504] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.229388 28361 leader_election.cc:288] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [CANDIDATE]: Term 1 election: Requested vote from peers 484aa4de22bf4443ad1cc14ec37366be (127.19.185.194:45653), b057f1ce479845f5b29f1bd34f6f7284 (127.19.185.195:45975)
I20260926 02:53:38.229831 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ced9a2d4b424cab10252f695327b0" candidate_uuid: "05960a65b34b4cd7ad9f4a61855240d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b057f1ce479845f5b29f1bd34f6f7284"
I20260926 02:53:38.229892 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ced9a2d4b424cab10252f695327b0" candidate_uuid: "05960a65b34b4cd7ad9f4a61855240d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484aa4de22bf4443ad1cc14ec37366be"
I20260926 02:53:38.230057 28280 raft_consensus.cc:3037] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:38.230075 28137 raft_consensus.cc:3037] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:38.232159 28137 raft_consensus.cc:2454] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05960a65b34b4cd7ad9f4a61855240d3 in term 1.
I20260926 02:53:38.232219 28280 raft_consensus.cc:2454] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05960a65b34b4cd7ad9f4a61855240d3 in term 1.
I20260926 02:53:38.232594 27929 leader_election.cc:302] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05960a65b34b4cd7ad9f4a61855240d3, 484aa4de22bf4443ad1cc14ec37366be; no voters:
I20260926 02:53:38.232846 28361 raft_consensus.cc:2781] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:38.233024 28361 raft_consensus.cc:686] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 LEADER]: Becoming Leader. State: Replica: 05960a65b34b4cd7ad9f4a61855240d3, State: Running, Role: LEADER
I20260926 02:53:38.233294 28361 consensus_queue.cc:237] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.234791 28356 tablet_replica.cc:439] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3: TxnStatusTablet state changed. Reason: New leader 05960a65b34b4cd7ad9f4a61855240d3. Latest consensus state: current_term: 1 leader_uuid: "05960a65b34b4cd7ad9f4a61855240d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } }
I20260926 02:53:38.235009 28356 tablet_replica.cc:442] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:53:38.235579 28394 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 02:53:38.235808 28394 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 02:53:38.236301 27841 catalog_manager.cc:5495] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05960a65b34b4cd7ad9f4a61855240d3 (127.19.185.193). New cstate: current_term: 1 leader_uuid: "05960a65b34b4cd7ad9f4a61855240d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } health_report { overall_health: UNKNOWN } } }
I20260926 02:53:38.261620 28348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.255630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.19.185.193:39795)
0926 02:53:38.255860 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:38.255866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:38.255886 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:53:38.256081 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:38.256085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:38.256196 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:38.256270 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:53:38.256685 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.257623 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.259675 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.260214 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.260325 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.260879 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:53:38.261032 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:53:38.261111 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:38.261304 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 02:53:38.261368 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:53:38.261706 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.255490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:39795 (local address 127.0.0.1:59454)
0926 02:53:38.255738 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:38.255759 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:38.255898 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:38.256331 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:38.256337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:38.256350 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:53:38.256547 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:38.256557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.257762 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.257768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.259525 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:38.259540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.260396 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.260404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.260613 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.260691 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:53:38.261144 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:53:38.261172 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 02:53:38.261261 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 02:53:38.261413 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 02:53:38.265658 28280 raft_consensus.cc:1261] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Refusing update from remote peer 05960a65b34b4cd7ad9f4a61855240d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:38.265662 28137 raft_consensus.cc:1261] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Refusing update from remote peer 05960a65b34b4cd7ad9f4a61855240d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:38.266268 28356 consensus_queue.cc:1035] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:38.266767 28361 consensus_queue.cc:1035] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:38.272558 28358 tablet_replica.cc:439] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be: TxnStatusTablet state changed. Reason: New leader 05960a65b34b4cd7ad9f4a61855240d3. Latest consensus state: current_term: 1 leader_uuid: "05960a65b34b4cd7ad9f4a61855240d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } }
I20260926 02:53:38.272835 28358 tablet_replica.cc:442] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:53:38.277081 28358 tablet_replica.cc:439] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05960a65b34b4cd7ad9f4a61855240d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } }
I20260926 02:53:38.277473 28358 tablet_replica.cc:442] T 233ced9a2d4b424cab10252f695327b0 P 484aa4de22bf4443ad1cc14ec37366be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:53:38.284734 28357 tablet_replica.cc:439] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284: TxnStatusTablet state changed. Reason: New leader 05960a65b34b4cd7ad9f4a61855240d3. Latest consensus state: current_term: 1 leader_uuid: "05960a65b34b4cd7ad9f4a61855240d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } }
I20260926 02:53:38.285006 28357 tablet_replica.cc:442] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:53:38.287395 28397 mvcc.cc:205] Tried to move back new op lower bound from 7333442430010122240 to 7333442429885394944. Current Snapshot: MvccSnapshot[applied={T|T < 7333442430010122240}]
I20260926 02:53:38.287851 28356 tablet_replica.cc:439] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05960a65b34b4cd7ad9f4a61855240d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } }
I20260926 02:53:38.288196 28356 tablet_replica.cc:442] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:53:38.288061 28361 tablet_replica.cc:439] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05960a65b34b4cd7ad9f4a61855240d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } }
I20260926 02:53:38.288525 28361 tablet_replica.cc:442] T 233ced9a2d4b424cab10252f695327b0 P 05960a65b34b4cd7ad9f4a61855240d3: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:53:38.293159 28357 tablet_replica.cc:439] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05960a65b34b4cd7ad9f4a61855240d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } } }
I20260926 02:53:38.293402 28357 tablet_replica.cc:442] T 233ced9a2d4b424cab10252f695327b0 P b057f1ce479845f5b29f1bd34f6f7284: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:53:38.301362 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.294981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.19.185.194:45653)
0926 02:53:38.295768 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:38.295775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:38.296030 (+ 255us) server_negotiation.cc:407] Connection header received
0926 02:53:38.296272 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:38.296281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:38.296371 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:38.296549 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:53:38.296980 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.297791 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.299131 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.299619 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.299757 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.300209 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:53:38.300739 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:53:38.300897 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:38.300956 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 02:53:38.301044 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":492,"threads_started":1}
I20260926 02:53:38.325093 28348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.312433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:39607 (local address 127.19.185.193:39795)
0926 02:53:38.312641 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:38.312649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:38.316338 (+ 3689us) server_negotiation.cc:407] Connection header received
0926 02:53:38.316607 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:38.316614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:38.316717 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:38.316820 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:53:38.317642 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.318777 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.320919 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.321846 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.321944 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.323992 (+ 2048us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:53:38.324399 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:53:38.324506 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:38.324666 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 02:53:38.324755 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 02:53:38.325385 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.314366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:39795 (local address 127.19.185.194:39607)
0926 02:53:38.316234 (+ 1868us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:38.316254 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:38.316417 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:38.316858 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:38.316865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:38.316887 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:53:38.317477 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:38.317490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.318947 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.318953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.320719 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:38.320734 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.323482 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.323489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.323668 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.323745 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:53:38.324521 (+ 776us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:53:38.324532 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 02:53:38.324604 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 02:53:38.324755 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1660,"thread_start_us":1176,"threads_started":1}
I20260926 02:53:38.336040 28280 raft_consensus.cc:3037] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:38.336426 27998 raft_consensus.cc:1261] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Refusing update from remote peer 484aa4de22bf4443ad1cc14ec37366be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:38.337229 28358 consensus_queue.cc:1035] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [LEADER]: Connected to new peer: Peer: permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:38.339159 28280 raft_consensus.cc:1261] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Refusing update from remote peer 484aa4de22bf4443ad1cc14ec37366be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:38.339798 28358 consensus_queue.cc:1035] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:38.348733 28398 mvcc.cc:205] Tried to move back new op lower bound from 7333442430298181632 to 7333442428993314816. Current Snapshot: MvccSnapshot[applied={T|T < 7333442430298181632}]
I20260926 02:53:38.382148 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.375594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:44487 (local address 127.19.185.194:45653)
0926 02:53:38.375776 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:38.375783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:38.375982 (+ 199us) server_negotiation.cc:407] Connection header received
0926 02:53:38.376221 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:38.376227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:38.376303 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:38.376403 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:53:38.377077 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.378225 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.380302 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.380757 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.380884 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.381347 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:53:38.381449 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:53:38.381556 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:38.381724 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 02:53:38.381829 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:53:38.383001 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.375144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:45653 (local address 127.19.185.193:44487)
0926 02:53:38.375881 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:38.375900 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:38.376098 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:38.376417 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:38.376421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:38.376432 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:53:38.376859 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:38.376873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.378429 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.378433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.380111 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:38.380125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.380898 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.380904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.381105 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.381183 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:53:38.381557 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:53:38.381570 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 02:53:38.382411 (+ 841us) client_negotiation.cc:240] Negotiation successful
0926 02:53:38.382549 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":649,"threads_started":1}
I20260926 02:53:38.421500 27841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:38.421763 27841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:38.425190 27841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f: Sending DeleteTablet for 3 replicas of tablet 6bfb5018d5274ba58cc72a11e713a9b1
I20260926 02:53:38.426028 28117 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bfb5018d5274ba58cc72a11e713a9b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:38 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:56924
I20260926 02:53:38.426162 28255 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bfb5018d5274ba58cc72a11e713a9b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:38 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:49636
I20260926 02:53:38.426836 27978 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bfb5018d5274ba58cc72a11e713a9b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:38 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:59446
I20260926 02:53:38.441622 28420 tablet_replica.cc:330] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284: stopping tablet replica
I20260926 02:53:38.441877 28420 raft_consensus.cc:2229] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:38.442188 28420 raft_consensus.cc:2258] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:38.443470 28420 ts_tablet_manager.cc:1905] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:38.449612 28421 tablet_replica.cc:330] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3: stopping tablet replica
I20260926 02:53:38.450366 28419 tablet_replica.cc:330] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be: stopping tablet replica
I20260926 02:53:38.450503 28421 raft_consensus.cc:2229] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:38.450732 28420 ts_tablet_manager.cc:1918] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:53:38.450871 28420 log.cc:1196] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/6bfb5018d5274ba58cc72a11e713a9b1
I20260926 02:53:38.450976 28421 raft_consensus.cc:2258] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:38.451375 28420 ts_tablet_manager.cc:1939] T 6bfb5018d5274ba58cc72a11e713a9b1 P b057f1ce479845f5b29f1bd34f6f7284: Deleting consensus metadata
I20260926 02:53:38.452452 27827 catalog_manager.cc:4841] TS b057f1ce479845f5b29f1bd34f6f7284 (127.19.185.195:45975): tablet 6bfb5018d5274ba58cc72a11e713a9b1 (table test-table [id=7549f588bc0e43f498e5ba3c7ddddb5e]) successfully deleted
I20260926 02:53:38.451252 28419 raft_consensus.cc:2229] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:53:38.453789 28419 raft_consensus.cc:2258] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:38.452454 28421 ts_tablet_manager.cc:1905] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:38.460386 28419 ts_tablet_manager.cc:1905] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:38.462313 28421 ts_tablet_manager.cc:1918] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:53:38.462436 28421 log.cc:1196] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/6bfb5018d5274ba58cc72a11e713a9b1
I20260926 02:53:38.462878 28421 ts_tablet_manager.cc:1939] T 6bfb5018d5274ba58cc72a11e713a9b1 P 05960a65b34b4cd7ad9f4a61855240d3: Deleting consensus metadata
I20260926 02:53:38.463965 27826 catalog_manager.cc:4841] TS 05960a65b34b4cd7ad9f4a61855240d3 (127.19.185.193:39795): tablet 6bfb5018d5274ba58cc72a11e713a9b1 (table test-table [id=7549f588bc0e43f498e5ba3c7ddddb5e]) successfully deleted
I20260926 02:53:38.470232 28419 ts_tablet_manager.cc:1918] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:53:38.475399 28419 log.cc:1196] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/6bfb5018d5274ba58cc72a11e713a9b1
I20260926 02:53:38.475982 28419 ts_tablet_manager.cc:1939] T 6bfb5018d5274ba58cc72a11e713a9b1 P 484aa4de22bf4443ad1cc14ec37366be: Deleting consensus metadata
I20260926 02:53:38.481020 27826 catalog_manager.cc:4841] TS 484aa4de22bf4443ad1cc14ec37366be (127.19.185.194:45653): tablet 6bfb5018d5274ba58cc72a11e713a9b1 (table test-table [id=7549f588bc0e43f498e5ba3c7ddddb5e]) successfully deleted
Sep 26 02:53:38 dist-test-slave-ktng krb5kdc[27796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:53:38 dist-test-slave-ktng krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:38 dist-test-slave-ktng krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:53:38.552213 28388 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:38.538818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.19.185.254:37991)
0926 02:53:38.538999 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:38.539006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:38.539522 (+ 516us) server_negotiation.cc:407] Connection header received
0926 02:53:38.539710 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:38.539715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:38.539812 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:38.539923 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:38.541706 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.542860 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.544767 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.545448 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.545578 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.551932 (+ 6354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34302: BlockingRecv error: recv got EOF from 127.0.0.1:34302 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 02:53:38 dist-test-slave-ktng krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM
I20260926 02:53:38.575495 28388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.557463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.19.185.254:37991)
0926 02:53:38.557774 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:38.557780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:38.557800 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:53:38.557960 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:38.557967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:38.558046 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:38.558158 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:38.559852 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.560984 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.562792 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.563474 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.563611 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.568610 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:38.568620 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:38.568624 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:38.568670 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:38.572340 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:38.573194 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:38.573203 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:38.573208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:38.573367 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:38.573846 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:38.573851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:38.573855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:38.574262 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:38.574536 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:38.574851 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 02:53:38.575091 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260926 02:53:38.579809 27841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:53:38.580521 27841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:53:38.588508 27978 tablet_service.cc:1467] Processing CreateTablet for tablet 583ca1f4f7a34872a831ba27666aa248 (DEFAULT_TABLE table=test-table [id=a82f408ac1eb4fca8a2066452819b1ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:38.588627 28255 tablet_service.cc:1467] Processing CreateTablet for tablet 583ca1f4f7a34872a831ba27666aa248 (DEFAULT_TABLE table=test-table [id=a82f408ac1eb4fca8a2066452819b1ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:38.588984 27978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583ca1f4f7a34872a831ba27666aa248. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:38.589107 28255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583ca1f4f7a34872a831ba27666aa248. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:38.591914 28117 tablet_service.cc:1467] Processing CreateTablet for tablet 583ca1f4f7a34872a831ba27666aa248 (DEFAULT_TABLE table=test-table [id=a82f408ac1eb4fca8a2066452819b1ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:38.592319 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583ca1f4f7a34872a831ba27666aa248. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:38.596485 28352 tablet_bootstrap.cc:492] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284: Bootstrap starting.
I20260926 02:53:38.597769 28352 tablet_bootstrap.cc:654] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:38.603662 28351 tablet_bootstrap.cc:492] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be: Bootstrap starting.
I20260926 02:53:38.604023 28350 tablet_bootstrap.cc:492] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3: Bootstrap starting.
I20260926 02:53:38.605463 28351 tablet_bootstrap.cc:654] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:38.605741 28350 tablet_bootstrap.cc:654] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:38.609869 28352 tablet_bootstrap.cc:492] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284: No bootstrap required, opened a new log
I20260926 02:53:38.609887 28351 tablet_bootstrap.cc:492] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be: No bootstrap required, opened a new log
I20260926 02:53:38.610025 28352 ts_tablet_manager.cc:1397] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.001s
I20260926 02:53:38.610037 28351 ts_tablet_manager.cc:1397] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260926 02:53:38.611954 28351 raft_consensus.cc:348] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.612227 28351 raft_consensus.cc:374] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:38.612327 28351 raft_consensus.cc:729] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484aa4de22bf4443ad1cc14ec37366be, State: Initialized, Role: FOLLOWER
I20260926 02:53:38.612334 28352 raft_consensus.cc:348] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.612599 28352 raft_consensus.cc:374] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:38.612691 28352 raft_consensus.cc:729] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b057f1ce479845f5b29f1bd34f6f7284, State: Initialized, Role: FOLLOWER
I20260926 02:53:38.612605 28351 consensus_queue.cc:260] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.613143 28351 ts_tablet_manager.cc:1428] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 02:53:38.613427 28352 consensus_queue.cc:260] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.614005 28352 ts_tablet_manager.cc:1428] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260926 02:53:38.616259 28350 tablet_bootstrap.cc:492] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3: No bootstrap required, opened a new log
I20260926 02:53:38.616427 28350 ts_tablet_manager.cc:1397] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260926 02:53:38.617240 28350 raft_consensus.cc:348] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.617501 28350 raft_consensus.cc:374] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:38.617591 28350 raft_consensus.cc:729] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05960a65b34b4cd7ad9f4a61855240d3, State: Initialized, Role: FOLLOWER
I20260926 02:53:38.617798 28350 consensus_queue.cc:260] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.618366 28350 ts_tablet_manager.cc:1428] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 02:53:38.636370 28357 raft_consensus.cc:482] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:53:38.636626 28357 raft_consensus.cc:504] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.638432 28357 leader_election.cc:288] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05960a65b34b4cd7ad9f4a61855240d3 (127.19.185.193:39795), 484aa4de22bf4443ad1cc14ec37366be (127.19.185.194:45653)
I20260926 02:53:38.647369 28348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.639015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:55929 (local address 127.19.185.193:39795)
0926 02:53:38.639210 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:38.639218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:38.639570 (+ 352us) server_negotiation.cc:407] Connection header received
0926 02:53:38.639843 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:38.639852 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:38.639924 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:38.639985 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:38.640601 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.641807 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.644883 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.646185 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.646330 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.646402 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:38.646926 (+ 524us) server_negotiation.cc:299] Negotiation successful
0926 02:53:38.647071 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:53:38.648087 28437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.638649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:39795 (local address 127.19.185.195:55929)
0926 02:53:38.639468 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:38.639494 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:38.639687 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:38.640175 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:38.640179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:38.640205 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:38.640422 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:38.640433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.642464 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.642469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.644688 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:38.644704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.646556 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.646563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.646742 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.646777 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:53:38.647626 (+ 849us) client_negotiation.cc:240] Negotiation successful
0926 02:53:38.647768 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":1251,"threads_started":1}
I20260926 02:53:38.652251 27998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583ca1f4f7a34872a831ba27666aa248" candidate_uuid: "b057f1ce479845f5b29f1bd34f6f7284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05960a65b34b4cd7ad9f4a61855240d3" is_pre_election: true
I20260926 02:53:38.652534 27998 raft_consensus.cc:2454] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b057f1ce479845f5b29f1bd34f6f7284 in term 0.
I20260926 02:53:38.653927 28208 leader_election.cc:302] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05960a65b34b4cd7ad9f4a61855240d3, b057f1ce479845f5b29f1bd34f6f7284; no voters:
I20260926 02:53:38.654578 28357 raft_consensus.cc:2781] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:53:38.654702 28357 raft_consensus.cc:482] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:53:38.654793 28357 raft_consensus.cc:3037] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:38.655834 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.642130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:45141 (local address 127.19.185.194:45653)
0926 02:53:38.642341 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:38.642347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:38.648398 (+ 6051us) server_negotiation.cc:407] Connection header received
0926 02:53:38.648721 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:38.648727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:38.648826 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:38.648887 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:53:38.649382 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.650610 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.653377 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.654649 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.654780 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.654853 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:38.655388 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 02:53:38.655541 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 02:53:38.656592 28437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.640339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:45653 (local address 127.19.185.195:45141)
0926 02:53:38.648313 (+ 7974us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:38.648331 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:38.648554 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:38.649055 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:38.649059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:38.649082 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:53:38.649225 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:38.649235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.651226 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.651231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.653199 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:38.653214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.654983 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.654989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:38.655182 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.655216 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:53:38.656056 (+ 840us) client_negotiation.cc:240] Negotiation successful
0926 02:53:38.656212 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7877,"thread_start_us":11116,"threads_started":1}
I20260926 02:53:38.657750 28136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583ca1f4f7a34872a831ba27666aa248" candidate_uuid: "b057f1ce479845f5b29f1bd34f6f7284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484aa4de22bf4443ad1cc14ec37366be" is_pre_election: true
I20260926 02:53:38.658071 28136 raft_consensus.cc:2454] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b057f1ce479845f5b29f1bd34f6f7284 in term 0.
I20260926 02:53:38.658113 28357 raft_consensus.cc:504] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.659585 28357 leader_election.cc:288] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [CANDIDATE]: Term 1 election: Requested vote from peers 05960a65b34b4cd7ad9f4a61855240d3 (127.19.185.193:39795), 484aa4de22bf4443ad1cc14ec37366be (127.19.185.194:45653)
I20260926 02:53:38.660156 27998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583ca1f4f7a34872a831ba27666aa248" candidate_uuid: "b057f1ce479845f5b29f1bd34f6f7284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05960a65b34b4cd7ad9f4a61855240d3"
I20260926 02:53:38.660384 27998 raft_consensus.cc:3037] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:38.660470 28136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583ca1f4f7a34872a831ba27666aa248" candidate_uuid: "b057f1ce479845f5b29f1bd34f6f7284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484aa4de22bf4443ad1cc14ec37366be"
I20260926 02:53:38.660701 28136 raft_consensus.cc:3037] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:38.662773 27998 raft_consensus.cc:2454] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b057f1ce479845f5b29f1bd34f6f7284 in term 1.
I20260926 02:53:38.663276 28208 leader_election.cc:302] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05960a65b34b4cd7ad9f4a61855240d3, b057f1ce479845f5b29f1bd34f6f7284; no voters:
I20260926 02:53:38.663600 28357 raft_consensus.cc:2781] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:38.663864 28136 raft_consensus.cc:2454] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b057f1ce479845f5b29f1bd34f6f7284 in term 1.
I20260926 02:53:38.664659 28357 raft_consensus.cc:686] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 LEADER]: Becoming Leader. State: Replica: b057f1ce479845f5b29f1bd34f6f7284, State: Running, Role: LEADER
I20260926 02:53:38.665096 28357 consensus_queue.cc:237] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.668843 27840 catalog_manager.cc:5495] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 reported cstate change: term changed from 0 to 1, leader changed from <none> to b057f1ce479845f5b29f1bd34f6f7284 (127.19.185.195). New cstate: current_term: 1 leader_uuid: "b057f1ce479845f5b29f1bd34f6f7284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } health_report { overall_health: HEALTHY } } }
I20260926 02:53:38.702876 28442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.694609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49652 (local address 127.19.185.195:45975)
0926 02:53:38.695709 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:38.695715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:38.695741 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:53:38.695862 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:38.695869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:38.695985 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:38.696190 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:53:38.696632 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.697834 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.699843 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.700510 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.700648 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.701183 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:53:38.701998 (+ 815us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:53:38.702164 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:38.702252 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 02:53:38.702334 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":798,"threads_started":1}
I20260926 02:53:38.707583 27998 raft_consensus.cc:1261] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Refusing update from remote peer b057f1ce479845f5b29f1bd34f6f7284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:38.707943 28136 raft_consensus.cc:1261] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Refusing update from remote peer b057f1ce479845f5b29f1bd34f6f7284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:38.708817 28357 consensus_queue.cc:1035] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:38.709405 28439 consensus_queue.cc:1035] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:38.756088 27840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:38.756340 27840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:38.759835 27840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f: Sending DeleteTablet for 3 replicas of tablet 583ca1f4f7a34872a831ba27666aa248
I20260926 02:53:38.760604 27978 tablet_service.cc:1514] Processing DeleteTablet for tablet 583ca1f4f7a34872a831ba27666aa248 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:38 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:59446
I20260926 02:53:38.760910 28421 tablet_replica.cc:330] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3: stopping tablet replica
I20260926 02:53:38.761116 28421 raft_consensus.cc:2229] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:38.761343 28421 raft_consensus.cc:2258] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:38.762743 28421 ts_tablet_manager.cc:1905] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:38.768206 28117 tablet_service.cc:1514] Processing DeleteTablet for tablet 583ca1f4f7a34872a831ba27666aa248 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:38 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:56924
I20260926 02:53:38.768519 28419 tablet_replica.cc:330] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be: stopping tablet replica
I20260926 02:53:38.768733 28419 raft_consensus.cc:2229] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:38.768973 28419 raft_consensus.cc:2258] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:38.770169 28419 ts_tablet_manager.cc:1905] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:38.776261 28255 tablet_service.cc:1514] Processing DeleteTablet for tablet 583ca1f4f7a34872a831ba27666aa248 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:38 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:49636
I20260926 02:53:38.776609 28420 tablet_replica.cc:330] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284: stopping tablet replica
I20260926 02:53:38.776818 28420 raft_consensus.cc:2229] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:53:38.777354 28420 raft_consensus.cc:2258] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:38.778756 28420 ts_tablet_manager.cc:1905] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:38.783035 27841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:53:38.783798 27841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:53:38.793881 27978 tablet_service.cc:1467] Processing CreateTablet for tablet ce3ea8e206094872a4a1d1d95ea468c7 (DEFAULT_TABLE table=test-table [id=f0fd2a9c3d9b4e13a828788701f647d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:38.794378 27978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3ea8e206094872a4a1d1d95ea468c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:38.799733 28117 tablet_service.cc:1467] Processing CreateTablet for tablet ce3ea8e206094872a4a1d1d95ea468c7 (DEFAULT_TABLE table=test-table [id=f0fd2a9c3d9b4e13a828788701f647d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:38.800184 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3ea8e206094872a4a1d1d95ea468c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:38.804365 28255 tablet_service.cc:1467] Processing CreateTablet for tablet ce3ea8e206094872a4a1d1d95ea468c7 (DEFAULT_TABLE table=test-table [id=f0fd2a9c3d9b4e13a828788701f647d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:53:38.804770 28255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3ea8e206094872a4a1d1d95ea468c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:38.817718 28420 ts_tablet_manager.cc:1918] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:38.823742 28351 tablet_bootstrap.cc:492] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be: Bootstrap starting.
I20260926 02:53:38.825543 28351 tablet_bootstrap.cc:654] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:38.832861 28350 tablet_bootstrap.cc:492] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3: Bootstrap starting.
I20260926 02:53:38.834602 28350 tablet_bootstrap.cc:654] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:38.835505 28419 ts_tablet_manager.cc:1918] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:38.835656 28419 log.cc:1196] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/583ca1f4f7a34872a831ba27666aa248
I20260926 02:53:38.836114 28419 ts_tablet_manager.cc:1939] T 583ca1f4f7a34872a831ba27666aa248 P 484aa4de22bf4443ad1cc14ec37366be: Deleting consensus metadata
I20260926 02:53:38.837400 27826 catalog_manager.cc:4841] TS 484aa4de22bf4443ad1cc14ec37366be (127.19.185.194:45653): tablet 583ca1f4f7a34872a831ba27666aa248 (table test-table [id=a82f408ac1eb4fca8a2066452819b1ce]) successfully deleted
I20260926 02:53:38.834602 28421 ts_tablet_manager.cc:1918] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:53:38.839105 28421 log.cc:1196] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/583ca1f4f7a34872a831ba27666aa248
I20260926 02:53:38.839810 28421 ts_tablet_manager.cc:1939] T 583ca1f4f7a34872a831ba27666aa248 P 05960a65b34b4cd7ad9f4a61855240d3: Deleting consensus metadata
I20260926 02:53:38.829898 28352 tablet_bootstrap.cc:492] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284: Bootstrap starting.
I20260926 02:53:38.840339 28420 log.cc:1196] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/583ca1f4f7a34872a831ba27666aa248
I20260926 02:53:38.840831 28420 ts_tablet_manager.cc:1939] T 583ca1f4f7a34872a831ba27666aa248 P b057f1ce479845f5b29f1bd34f6f7284: Deleting consensus metadata
I20260926 02:53:38.842011 28352 tablet_bootstrap.cc:654] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:38.842123 27827 catalog_manager.cc:4841] TS b057f1ce479845f5b29f1bd34f6f7284 (127.19.185.195:45975): tablet 583ca1f4f7a34872a831ba27666aa248 (table test-table [id=a82f408ac1eb4fca8a2066452819b1ce]) successfully deleted
I20260926 02:53:38.845374 28351 tablet_bootstrap.cc:492] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be: No bootstrap required, opened a new log
I20260926 02:53:38.845558 28351 ts_tablet_manager.cc:1397] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.005s
I20260926 02:53:38.846496 28351 raft_consensus.cc:348] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.846772 28351 raft_consensus.cc:374] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:38.846866 28351 raft_consensus.cc:729] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484aa4de22bf4443ad1cc14ec37366be, State: Initialized, Role: FOLLOWER
I20260926 02:53:38.847150 28351 consensus_queue.cc:260] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.847887 28351 ts_tablet_manager.cc:1428] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260926 02:53:38.849783 28350 tablet_bootstrap.cc:492] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3: No bootstrap required, opened a new log
I20260926 02:53:38.850337 28350 ts_tablet_manager.cc:1397] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.005s
I20260926 02:53:38.850512 27826 catalog_manager.cc:4841] TS 05960a65b34b4cd7ad9f4a61855240d3 (127.19.185.193:39795): tablet 583ca1f4f7a34872a831ba27666aa248 (table test-table [id=a82f408ac1eb4fca8a2066452819b1ce]) successfully deleted
I20260926 02:53:38.852345 28350 raft_consensus.cc:348] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.853040 28350 raft_consensus.cc:374] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:38.853143 28350 raft_consensus.cc:729] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05960a65b34b4cd7ad9f4a61855240d3, State: Initialized, Role: FOLLOWER
I20260926 02:53:38.853381 28350 consensus_queue.cc:260] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.853928 28350 ts_tablet_manager.cc:1428] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 02:53:38.857396 28352 tablet_bootstrap.cc:492] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284: No bootstrap required, opened a new log
I20260926 02:53:38.857606 28352 ts_tablet_manager.cc:1397] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.009s
I20260926 02:53:38.858624 28352 raft_consensus.cc:348] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.858906 28352 raft_consensus.cc:374] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:38.859004 28352 raft_consensus.cc:729] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b057f1ce479845f5b29f1bd34f6f7284, State: Initialized, Role: FOLLOWER
I20260926 02:53:38.859252 28352 consensus_queue.cc:260] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.861191 28352 ts_tablet_manager.cc:1428] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 02:53:38.897748 28416 raft_consensus.cc:482] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:53:38.898013 28416 raft_consensus.cc:504] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.899103 28416 leader_election.cc:288] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05960a65b34b4cd7ad9f4a61855240d3 (127.19.185.193:39795), b057f1ce479845f5b29f1bd34f6f7284 (127.19.185.195:45975)
I20260926 02:53:38.899595 28281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3ea8e206094872a4a1d1d95ea468c7" candidate_uuid: "484aa4de22bf4443ad1cc14ec37366be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b057f1ce479845f5b29f1bd34f6f7284" is_pre_election: true
I20260926 02:53:38.899848 28281 raft_consensus.cc:2454] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484aa4de22bf4443ad1cc14ec37366be in term 0.
I20260926 02:53:38.900283 27998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3ea8e206094872a4a1d1d95ea468c7" candidate_uuid: "484aa4de22bf4443ad1cc14ec37366be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05960a65b34b4cd7ad9f4a61855240d3" is_pre_election: true
I20260926 02:53:38.900439 28070 leader_election.cc:302] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484aa4de22bf4443ad1cc14ec37366be, b057f1ce479845f5b29f1bd34f6f7284; no voters:
I20260926 02:53:38.900530 27998 raft_consensus.cc:2454] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484aa4de22bf4443ad1cc14ec37366be in term 0.
I20260926 02:53:38.900739 28416 raft_consensus.cc:2781] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:53:38.900854 28416 raft_consensus.cc:482] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:53:38.900939 28416 raft_consensus.cc:3037] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:38.904182 28416 raft_consensus.cc:504] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.905277 28416 leader_election.cc:288] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [CANDIDATE]: Term 1 election: Requested vote from peers 05960a65b34b4cd7ad9f4a61855240d3 (127.19.185.193:39795), b057f1ce479845f5b29f1bd34f6f7284 (127.19.185.195:45975)
I20260926 02:53:38.905292 27998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3ea8e206094872a4a1d1d95ea468c7" candidate_uuid: "484aa4de22bf4443ad1cc14ec37366be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05960a65b34b4cd7ad9f4a61855240d3"
I20260926 02:53:38.905558 27998 raft_consensus.cc:3037] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:38.905644 28281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3ea8e206094872a4a1d1d95ea468c7" candidate_uuid: "484aa4de22bf4443ad1cc14ec37366be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b057f1ce479845f5b29f1bd34f6f7284"
I20260926 02:53:38.905872 28281 raft_consensus.cc:3037] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:38.908792 27998 raft_consensus.cc:2454] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484aa4de22bf4443ad1cc14ec37366be in term 1.
I20260926 02:53:38.909277 28281 raft_consensus.cc:2454] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484aa4de22bf4443ad1cc14ec37366be in term 1.
I20260926 02:53:38.909332 28069 leader_election.cc:302] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05960a65b34b4cd7ad9f4a61855240d3, 484aa4de22bf4443ad1cc14ec37366be; no voters:
I20260926 02:53:38.909814 28416 raft_consensus.cc:2781] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:38.910454 28416 raft_consensus.cc:686] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [term 1 LEADER]: Becoming Leader. State: Replica: 484aa4de22bf4443ad1cc14ec37366be, State: Running, Role: LEADER
I20260926 02:53:38.910746 28416 consensus_queue.cc:237] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } }
I20260926 02:53:38.913658 27841 catalog_manager.cc:5495] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be reported cstate change: term changed from 0 to 1, leader changed from <none> to 484aa4de22bf4443ad1cc14ec37366be (127.19.185.194). New cstate: current_term: 1 leader_uuid: "484aa4de22bf4443ad1cc14ec37366be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "484aa4de22bf4443ad1cc14ec37366be" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 45653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 } health_report { overall_health: UNKNOWN } } }
I20260926 02:53:38.948940 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:38.940976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.19.185.194:45653)
0926 02:53:38.941204 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:38.941212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:38.941295 (+ 83us) server_negotiation.cc:407] Connection header received
0926 02:53:38.941553 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:38.941559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:38.941653 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:38.941764 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:53:38.942367 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.943722 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.945965 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:38.946822 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:38.946966 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:38.947659 (+ 693us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:53:38.947825 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:53:38.947912 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:38.948469 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 02:53:38.948567 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:53:38.986256 27998 raft_consensus.cc:1261] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Refusing update from remote peer 484aa4de22bf4443ad1cc14ec37366be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:38.987259 28449 consensus_queue.cc:1035] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [LEADER]: Connected to new peer: Peer: permanent_uuid: "05960a65b34b4cd7ad9f4a61855240d3" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 39795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:38.987668 28281 raft_consensus.cc:1261] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Refusing update from remote peer 484aa4de22bf4443ad1cc14ec37366be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:53:38.988314 28449 consensus_queue.cc:1035] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b057f1ce479845f5b29f1bd34f6f7284" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:53:39.077201 27840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:39.077467 27840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:53:39.080415 27840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c907464d33f54617ae3856d4f75cab9f: Sending DeleteTablet for 3 replicas of tablet ce3ea8e206094872a4a1d1d95ea468c7
I20260926 02:53:39.081233 27978 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3ea8e206094872a4a1d1d95ea468c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:39 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:59446
I20260926 02:53:39.081267 28117 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3ea8e206094872a4a1d1d95ea468c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:39 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:56924
I20260926 02:53:39.081518 28419 tablet_replica.cc:330] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be: stopping tablet replica
I20260926 02:53:39.081568 28421 tablet_replica.cc:330] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3: stopping tablet replica
I20260926 02:53:39.081713 28419 raft_consensus.cc:2229] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:53:39.081737 28255 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3ea8e206094872a4a1d1d95ea468c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:53:39 UTC) from {username='oryx', principal='oryx/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:49636
I20260926 02:53:39.081785 28421 raft_consensus.cc:2229] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:39.082077 28419 raft_consensus.cc:2258] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:39.082086 28421 raft_consensus.cc:2258] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:39.082403 28420 tablet_replica.cc:330] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284: stopping tablet replica
I20260926 02:53:39.082612 28420 raft_consensus.cc:2229] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:53:39.082850 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 27912
I20260926 02:53:39.082877 28420 raft_consensus.cc:2258] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:53:39.084954 28420 ts_tablet_manager.cc:1905] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:39.088100 28419 ts_tablet_manager.cc:1905] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:39.094301 28419 ts_tablet_manager.cc:1918] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:53:39.094535 28419 log.cc:1196] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/ce3ea8e206094872a4a1d1d95ea468c7
I20260926 02:53:39.095044 28419 ts_tablet_manager.cc:1939] T ce3ea8e206094872a4a1d1d95ea468c7 P 484aa4de22bf4443ad1cc14ec37366be: Deleting consensus metadata
Sep 26 02:53:39 dist-test-slave-ktng krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.185.193@KRBTEST.COM
I20260926 02:53:39.096124 28420 ts_tablet_manager.cc:1918] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:53:39.096269 28420 log.cc:1196] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/ce3ea8e206094872a4a1d1d95ea468c7
I20260926 02:53:39.096371 27826 catalog_manager.cc:4841] TS 484aa4de22bf4443ad1cc14ec37366be (127.19.185.194:45653): tablet ce3ea8e206094872a4a1d1d95ea468c7 (table test-table [id=f0fd2a9c3d9b4e13a828788701f647d0]) successfully deleted
I20260926 02:53:39.096709 28420 ts_tablet_manager.cc:1939] T ce3ea8e206094872a4a1d1d95ea468c7 P b057f1ce479845f5b29f1bd34f6f7284: Deleting consensus metadata
I20260926 02:53:39.097822 27827 catalog_manager.cc:4841] TS b057f1ce479845f5b29f1bd34f6f7284 (127.19.185.195:45975): tablet ce3ea8e206094872a4a1d1d95ea468c7 (table test-table [id=f0fd2a9c3d9b4e13a828788701f647d0]) successfully deleted
I20260926 02:53:39.099200 28421 ts_tablet_manager.cc:1905] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:53:39.104451 28348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:39.084093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.19.185.193:39795)
0926 02:53:39.084288 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:39.084294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:39.085722 (+ 1428us) server_negotiation.cc:407] Connection header received
0926 02:53:39.085907 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:39.085912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:39.085993 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:39.086089 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:39.087811 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:39.088871 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:39.090802 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:39.091456 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:39.091580 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:39.096903 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:39.096928 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:39.096942 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:39.096999 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:39.100978 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:39.101922 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:39.101932 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:39.101936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:39.102134 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:39.102592 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:39.102600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:39.102606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:39.103132 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:39.103534 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:39.103786 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 02:53:39.104043 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:53:39.105026 28421 ts_tablet_manager.cc:1918] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:53:39.105136 28043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:39.105175 28421 log.cc:1196] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/ce3ea8e206094872a4a1d1d95ea468c7
I20260926 02:53:39.285578 28421 ts_tablet_manager.cc:1939] T ce3ea8e206094872a4a1d1d95ea468c7 P 05960a65b34b4cd7ad9f4a61855240d3: Deleting consensus metadata
I20260926 02:53:39.285734 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 27912
I20260926 02:53:39.312042 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 28052
W20260926 02:53:39.312073 27826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.185.193:39795 (error 108)
I20260926 02:53:39.314638 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:39.313840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:39795 (local address 127.0.0.1:59474)
0926 02:53:39.314369 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:39.314468 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":267,"threads_started":1}
W20260926 02:53:39.314901 27826 catalog_manager.cc:4584] TS 05960a65b34b4cd7ad9f4a61855240d3 (127.19.185.193:39795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce3ea8e206094872a4a1d1d95ea468c7: Network error: Client connection negotiation failed: client connection to 127.19.185.193:39795: connect: Connection refused (error 111)
Sep 26 02:53:39 dist-test-slave-ktng krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.185.194@KRBTEST.COM
I20260926 02:53:39.329509 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:39.312897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56944 (local address 127.19.185.194:45653)
0926 02:53:39.313091 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:39.313097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:39.313447 (+ 350us) server_negotiation.cc:407] Connection header received
0926 02:53:39.313526 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:39.313530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:39.313598 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:39.313726 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:39.315475 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:39.316281 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:39.318173 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:39.318604 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:39.318698 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:39.323592 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:39.323607 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:39.323621 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:39.323659 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:39.326623 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:39.327425 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:39.327433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:39.327435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:39.327565 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:39.327934 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:39.327940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:39.327943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:39.328323 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:39.328634 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:39.328958 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 02:53:39.329184 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:53:39.330199 28182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:39.368516 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:39.367970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:39795 (local address 127.0.0.1:59476)
0926 02:53:39.368224 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:39.368335 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 02:53:39.369653 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:39.369243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:39795 (local address 127.0.0.1:59492)
0926 02:53:39.369402 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:39.369483 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 02:53:39.430109 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:39.429536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:39795 (local address 127.0.0.1:59508)
0926 02:53:39.429787 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:39.429924 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 02:53:39.431257 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:39.430850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:39795 (local address 127.0.0.1:59520)
0926 02:53:39.431013 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:39.431096 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 02:53:39.456619 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 28052
I20260926 02:53:39.479274 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 28191
Sep 26 02:53:39 dist-test-slave-ktng krb5kdc[27796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.185.195@KRBTEST.COM
I20260926 02:53:39.495134 28465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:39.480188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.19.185.195:45975)
0926 02:53:39.480657 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:39.480661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:39.480677 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:53:39.480732 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:39.480736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:39.480795 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:39.480892 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:39.482127 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:39.482813 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:39.484091 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:39.484537 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:39.484644 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:39.489156 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:39.489172 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:39.489189 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:39.489238 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:39.492333 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:39.492977 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:39.492987 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:39.492990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:39.493169 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:39.493533 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:39.493538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:39.493541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:39.493967 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:39.494297 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:39.494457 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 02:53:39.494676 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":263,"threads_started":1}
I20260926 02:53:39.495864 28321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:39.512077 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:39.511612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:39795 (local address 127.0.0.1:59536)
0926 02:53:39.511829 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:39.511935 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 02:53:39.513217 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:39.512849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:39795 (local address 127.0.0.1:59540)
0926 02:53:39.513018 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:39.513083 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 02:53:39.618777 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 28191
I20260926 02:53:39.640839 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 27809
I20260926 02:53:39.642033 27881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:39.759362 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:39.758731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:39795 (local address 127.0.0.1:59542)
0926 02:53:39.759019 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:39.759131 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 02:53:39.759574 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 27809
2026-09-26T02:53:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5721 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:53:39 dist-test-slave-ktng krb5kdc[28470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:53:39 dist-test-slave-ktng krb5kdc[28470](info): set up 2 sockets
Sep 26 02:53:39 dist-test-slave-ktng krb5kdc[28470](info): commencing operation
krb5kdc: starting...
W20260926 02:53:40.865365 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:53:41 dist-test-slave-ktng krb5kdc[28470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:53:41.102705 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:44191
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:46789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:44191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:41.446770 28483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:41.447053 28483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:41.447162 28483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:41.454661 28483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:41.454771 28483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:41.454816 28483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:41.454869 28483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:41.454916 28483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:53:41.465020 28483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:46789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:44191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:44191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:41.466312 28483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:41.467658 28483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:41.479712 28491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:41.479774 28488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:41.479816 28489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:41.480856 28483 server_base.cc:1034] running on GCE node
I20260926 02:53:41.481786 28483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:41.483361 28483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:41.484651 28483 hybrid_clock.cc:648] HybridClock initialized: now 1790391221484595 us; error 54 us; skew 500 ppm
I20260926 02:53:41.485198 28483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T02:53:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1791 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:53:41 dist-test-slave-ktng krb5kdc[28497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:53:41 dist-test-slave-ktng krb5kdc[28497](info): set up 2 sockets
Sep 26 02:53:41 dist-test-slave-ktng krb5kdc[28497](info): commencing operation
krb5kdc: starting...
W20260926 02:53:42.657106 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:53:42 dist-test-slave-ktng krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:53:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:53:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:53:42.889195 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:39061
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:39061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:43.219470 28510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:43.219830 28510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:43.220005 28510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:43.227120 28510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:43.227276 28510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:43.227373 28510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:43.227479 28510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:43.227589 28510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:53:43.237241 28510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:39061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:39061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:43.238644 28510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:43.240032 28510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:43.252074 28518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:43.252154 28516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:43.252130 28515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:43.252792 28510 server_base.cc:1034] running on GCE node
I20260926 02:53:43.253618 28510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:43.255187 28510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:43.256489 28510 hybrid_clock.cc:648] HybridClock initialized: now 1790391223256404 us; error 54 us; skew 500 ppm
I20260926 02:53:43.256923 28510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:43 dist-test-slave-ktng krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391223, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:43.266060 28510 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:53:43.267848 28510 webserver.cc:460] Webserver started at http://127.19.185.254:36507/ using document root <none> and password file <none>
I20260926 02:53:43.268589 28510 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:43.268728 28510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:43.269059 28510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:43.272361 28510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "98534291b94942f8b86d68e94534e563"
format_stamp: "Formatted at 2026-09-26 02:53:43 on dist-test-slave-ktng"
server_key: "14bf92af1d4b618af618d604f766ec50"
server_key_iv: "96414a5a612fbff27221e66ceaf40816"
server_key_version: "encryptionkey@0"
I20260926 02:53:43.273272 28510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "98534291b94942f8b86d68e94534e563"
format_stamp: "Formatted at 2026-09-26 02:53:43 on dist-test-slave-ktng"
server_key: "14bf92af1d4b618af618d604f766ec50"
server_key_iv: "96414a5a612fbff27221e66ceaf40816"
server_key_version: "encryptionkey@0"
I20260926 02:53:43.278857 28510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:53:43.282790 28525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:43.284673 28510 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:53:43.284900 28510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "98534291b94942f8b86d68e94534e563"
format_stamp: "Formatted at 2026-09-26 02:53:43 on dist-test-slave-ktng"
server_key: "14bf92af1d4b618af618d604f766ec50"
server_key_iv: "96414a5a612fbff27221e66ceaf40816"
server_key_version: "encryptionkey@0"
I20260926 02:53:43.285130 28510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:43.302413 28510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:43.306847 28510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:43.307149 28510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:43.325978 28510 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:39061
I20260926 02:53:43.326001 28576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:39061 every 8 connection(s)
I20260926 02:53:43.327425 28510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:53:43.331194 28577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:53:43.331207 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 28510
I20260926 02:53:43.331440 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:53:43.332098 20199 external_mini_cluster.cc:1428] Setting key 3e95b88537614ba0dc32fc2edd4cc67a
I20260926 02:53:43.345789 28577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563: Bootstrap starting.
Sep 26 02:53:43 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:43.349620 28577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563: Neither blocks nor log segments found. Creating new log.
I20260926 02:53:43.350697 28577 log.cc:824] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:43.353701 28577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563: No bootstrap required, opened a new log
I20260926 02:53:43.357165 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:43.334375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.19.185.254:39061)
0926 02:53:43.335278 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:43.335347 (+ 69us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:43.335406 (+ 59us) server_negotiation.cc:407] Connection header received
0926 02:53:43.337728 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:43.337757 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:43.338226 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:43.338600 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:43.340475 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:43.342116 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:43.344122 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:43.344788 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:43.344924 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:43.349621 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:43.349628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:43.349643 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:43.349716 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:43.352863 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:43.353597 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:43.353604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:43.353610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:43.353855 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:43.354243 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:43.354248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:43.354256 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:43.354751 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:43.355075 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:43.355552 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 02:53:43.356072 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":420,"threads_started":1}
I20260926 02:53:43.362145 28577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98534291b94942f8b86d68e94534e563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39061 } }
I20260926 02:53:43.362536 28577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:53:43.362641 28577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98534291b94942f8b86d68e94534e563, State: Initialized, Role: FOLLOWER
I20260926 02:53:43.363212 28577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98534291b94942f8b86d68e94534e563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39061 } }
I20260926 02:53:43.363551 28577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:53:43.363692 28577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:53:43.363868 28577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:53:43.366045 28577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98534291b94942f8b86d68e94534e563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39061 } }
I20260926 02:53:43.366521 28577 leader_election.cc:302] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98534291b94942f8b86d68e94534e563; no voters:
I20260926 02:53:43.367095 28577 leader_election.cc:288] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:53:43.367233 28582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:53:43.367935 28582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 1 LEADER]: Becoming Leader. State: Replica: 98534291b94942f8b86d68e94534e563, State: Running, Role: LEADER
I20260926 02:53:43.368429 28582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98534291b94942f8b86d68e94534e563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39061 } }
I20260926 02:53:43.369174 28577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:43.371522 28584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98534291b94942f8b86d68e94534e563. Latest consensus state: current_term: 1 leader_uuid: "98534291b94942f8b86d68e94534e563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98534291b94942f8b86d68e94534e563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39061 } } }
I20260926 02:53:43.371317 28583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98534291b94942f8b86d68e94534e563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98534291b94942f8b86d68e94534e563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39061 } } }
I20260926 02:53:43.372217 28584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:43.372217 28583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:43.373314 28590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:53:43.379808 28590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:53:43.389313 28590 catalog_manager.cc:1347] Generated new cluster ID: 94376fc2b3e147bfa80175c67a1aeebf
I20260926 02:53:43.389420 28590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:53:43.401664 28590 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:53:43.403493 28590 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:53:43.423127 28590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563: Generated new TSK 0
I20260926 02:53:43.424281 28590 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:53:43.495393 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39061
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:43.833686 28603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:43.833999 28603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:43.834216 28603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:43.841526 28603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:43.841635 28603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:43.841851 28603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:53:43.851879 28603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:43.853264 28603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:43.854550 28603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:43.868798 28608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:43.868743 28609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:43.868920 28611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:43.869817 28603 server_base.cc:1034] running on GCE node
I20260926 02:53:43.870517 28603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:43.871565 28603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:43.872788 28603 hybrid_clock.cc:648] HybridClock initialized: now 1790391223872741 us; error 36 us; skew 500 ppm
I20260926 02:53:43.873183 28603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:43 dist-test-slave-ktng krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391223, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:43.877000 28603 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:53:43.878593 28603 webserver.cc:460] Webserver started at http://127.19.185.193:40359/ using document root <none> and password file <none>
I20260926 02:53:43.879422 28603 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:43.879577 28603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:43.879926 28603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:43.883093 28603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "48b6b24a3c4a4c15a9613df10b9e12c0"
format_stamp: "Formatted at 2026-09-26 02:53:43 on dist-test-slave-ktng"
server_key: "9b1064dedee7bcc3263f56cc349de996"
server_key_iv: "370fc3ad3ff0f604021000c2f6dda223"
server_key_version: "encryptionkey@0"
I20260926 02:53:43.884078 28603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "48b6b24a3c4a4c15a9613df10b9e12c0"
format_stamp: "Formatted at 2026-09-26 02:53:43 on dist-test-slave-ktng"
server_key: "9b1064dedee7bcc3263f56cc349de996"
server_key_iv: "370fc3ad3ff0f604021000c2f6dda223"
server_key_version: "encryptionkey@0"
I20260926 02:53:43.890050 28603 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:53:43.893841 28618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:43.895402 28603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 02:53:43.895601 28603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "48b6b24a3c4a4c15a9613df10b9e12c0"
format_stamp: "Formatted at 2026-09-26 02:53:43 on dist-test-slave-ktng"
server_key: "9b1064dedee7bcc3263f56cc349de996"
server_key_iv: "370fc3ad3ff0f604021000c2f6dda223"
server_key_version: "encryptionkey@0"
I20260926 02:53:43.895860 28603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:43.914465 28603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:43.919246 28603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:43.919597 28603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:43.920702 28603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:43.922130 28603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:43.922247 28603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:43.922420 28603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:43.922502 28603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:43.958317 28603 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:46339
I20260926 02:53:43.958364 28730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:46339 every 8 connection(s)
I20260926 02:53:43.959663 28603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:53:43.960044 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 28603
I20260926 02:53:43.960255 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:53:43.961108 20199 external_mini_cluster.cc:1428] Setting key b13a4ef4f4cd96e90c157ce61eb7c3bc
Sep 26 02:53:43 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391223, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:43.985116 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:43.962419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:59217 (local address 127.19.185.254:39061)
0926 02:53:43.962890 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:43.962896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:43.964179 (+ 1283us) server_negotiation.cc:407] Connection header received
0926 02:53:43.967026 (+ 2847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:43.967030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:43.967095 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:43.967205 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:43.969353 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:43.970252 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:43.972383 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:43.973181 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:43.973330 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:43.977380 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:43.977389 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:43.977397 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:43.977443 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:43.981231 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:43.982295 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:43.982304 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:43.982310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:43.982464 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:43.982920 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:43.982927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:43.982933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:43.983397 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:43.983570 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:43.984399 (+ 829us) server_negotiation.cc:299] Negotiation successful
0926 02:53:43.984654 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":235,"threads_started":1}
I20260926 02:53:43.985849 28734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:43.963115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:59217)
0926 02:53:43.963994 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:43.964047 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:43.966724 (+ 2677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:43.967504 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:43.967519 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:43.968302 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:43.969136 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:43.969166 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:43.970426 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:43.970435 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:43.972222 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:43.972235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:43.973407 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:43.973412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:43.973676 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:43.974383 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:43.974410 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:43.977057 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:43.981420 (+ 4363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:43.981428 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:43.981443 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:43.982135 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:43.982594 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:43.982598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:43.982600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:43.982781 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:43.983567 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:43.983574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:43.983934 (+ 360us) client_negotiation.cc:769] Sending connection context
0926 02:53:43.984248 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 02:53:43.984673 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":357,"threads_started":1}
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
I20260926 02:53:43.987741 28731 heartbeater.cc:347] Connected to a master server at 127.19.185.254:39061
I20260926 02:53:43.988155 28731 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:43.989245 28731 heartbeater.cc:511] Master 127.19.185.254:39061 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
I20260926 02:53:43.991994 28542 ts_manager.cc:194] Registered new tserver with Master: 48b6b24a3c4a4c15a9613df10b9e12c0 (127.19.185.193:46339)
I20260926 02:53:43.994169 28542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:59217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:53:44.019661 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39061
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:44.345065 28737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:44.345464 28737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:44.345639 28737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:44.352892 28737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:44.353045 28737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:44.353240 28737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:53:44.362942 28737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:44.364296 28737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:44.365646 28737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:44.379632 28742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:44.379695 28743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:44.379717 28745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:44.381423 28737 server_base.cc:1034] running on GCE node
I20260926 02:53:44.382023 28737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:44.382910 28737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:44.384109 28737 hybrid_clock.cc:648] HybridClock initialized: now 1790391224384065 us; error 36 us; skew 500 ppm
I20260926 02:53:44.384455 28737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:44 dist-test-slave-ktng krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391224, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:44.387570 28737 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:53:44.389436 28737 webserver.cc:460] Webserver started at http://127.19.185.194:35131/ using document root <none> and password file <none>
I20260926 02:53:44.390255 28737 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:44.390421 28737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:44.390738 28737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:44.393637 28737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "bb6e968404f64955b4f709f7c21bf089"
format_stamp: "Formatted at 2026-09-26 02:53:44 on dist-test-slave-ktng"
server_key: "b19beca6a0f4718e34d013add9c9737c"
server_key_iv: "d39bd18bae4cf237aad95699f8e60086"
server_key_version: "encryptionkey@0"
I20260926 02:53:44.394699 28737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "bb6e968404f64955b4f709f7c21bf089"
format_stamp: "Formatted at 2026-09-26 02:53:44 on dist-test-slave-ktng"
server_key: "b19beca6a0f4718e34d013add9c9737c"
server_key_iv: "d39bd18bae4cf237aad95699f8e60086"
server_key_version: "encryptionkey@0"
I20260926 02:53:44.400399 28737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260926 02:53:44.403854 28752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:44.405149 28737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:53:44.405306 28737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "bb6e968404f64955b4f709f7c21bf089"
format_stamp: "Formatted at 2026-09-26 02:53:44 on dist-test-slave-ktng"
server_key: "b19beca6a0f4718e34d013add9c9737c"
server_key_iv: "d39bd18bae4cf237aad95699f8e60086"
server_key_version: "encryptionkey@0"
I20260926 02:53:44.405500 28737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:44.422477 28737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:44.426815 28737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:44.427134 28737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:44.428313 28737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:44.429765 28737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:44.429845 28737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:44.430015 28737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:44.430090 28737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:44.464435 28737 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:36579
I20260926 02:53:44.464473 28864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:36579 every 8 connection(s)
I20260926 02:53:44.465837 28737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:53:44.471354 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 28737
I20260926 02:53:44.471527 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:53:44.472060 20199 external_mini_cluster.cc:1428] Setting key 9bb1c68c8ade5ba41efa3987f3e35956
Sep 26 02:53:44 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391224, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:44.489646 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:44.468586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:37789 (local address 127.19.185.254:39061)
0926 02:53:44.468774 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:44.468778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:44.470168 (+ 1390us) server_negotiation.cc:407] Connection header received
0926 02:53:44.472504 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:44.472509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:44.472588 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:44.472714 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:44.474499 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:44.475606 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:44.478186 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:44.478658 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:44.478762 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:44.482903 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:44.482908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:44.482911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:44.482942 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:44.485824 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:44.486871 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:44.486878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:44.486882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:44.487022 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:44.487612 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:44.487617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:44.487621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:44.487919 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:44.488100 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:44.489003 (+ 903us) server_negotiation.cc:299] Negotiation successful
0926 02:53:44.489268 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:53:44.490767 28867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:44.469191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:37789)
0926 02:53:44.469922 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:44.469959 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:44.472186 (+ 2227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:44.472909 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:44.472917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:44.473551 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:44.474280 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:44.474313 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:44.475756 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:44.475765 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:44.478026 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:44.478044 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:44.478892 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:44.478901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:44.479224 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:44.479978 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:44.480013 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:44.482631 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:44.486024 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:44.486035 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:44.486054 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:44.486733 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:44.487161 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:44.487166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:44.487171 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:44.487469 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:44.488062 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:44.488071 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:44.488469 (+ 398us) client_negotiation.cc:769] Sending connection context
0926 02:53:44.488894 (+ 425us) client_negotiation.cc:240] Negotiation successful
0926 02:53:44.489522 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":279,"threads_started":1}
I20260926 02:53:44.492403 28865 heartbeater.cc:347] Connected to a master server at 127.19.185.254:39061
I20260926 02:53:44.492753 28865 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:44.493530 28865 heartbeater.cc:511] Master 127.19.185.254:39061 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
I20260926 02:53:44.495203 28542 ts_manager.cc:194] Registered new tserver with Master: bb6e968404f64955b4f709f7c21bf089 (127.19.185.194:36579)
I20260926 02:53:44.496166 28542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:37789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:53:44.527674 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39061
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:44.854024 28870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:44.854315 28870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:44.854491 28870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:44.861918 28870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:44.862040 28870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:44.862241 28870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:53:44.872587 28870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:44.873908 28870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:44.875396 28870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:44.889335 28875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:44.889400 28876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:44.889364 28878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:44.890556 28870 server_base.cc:1034] running on GCE node
I20260926 02:53:44.891239 28870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:44.892280 28870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:44.893522 28870 hybrid_clock.cc:648] HybridClock initialized: now 1790391224893479 us; error 57 us; skew 500 ppm
I20260926 02:53:44.893849 28870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:44 dist-test-slave-ktng krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391224, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:44.897348 28870 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:53:44.898994 28870 webserver.cc:460] Webserver started at http://127.19.185.195:37129/ using document root <none> and password file <none>
I20260926 02:53:44.899813 28870 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:44.899968 28870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:44.900362 28870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:53:44.903434 28870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "c783db2d5df44d6a95fe9a73d04791c8"
format_stamp: "Formatted at 2026-09-26 02:53:44 on dist-test-slave-ktng"
server_key: "95d3eb240da1ae7e6ff87196fc93f23a"
server_key_iv: "d1bd01e3bf9e353918a941590fb0c78a"
server_key_version: "encryptionkey@0"
I20260926 02:53:44.904227 28870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "c783db2d5df44d6a95fe9a73d04791c8"
format_stamp: "Formatted at 2026-09-26 02:53:44 on dist-test-slave-ktng"
server_key: "95d3eb240da1ae7e6ff87196fc93f23a"
server_key_iv: "d1bd01e3bf9e353918a941590fb0c78a"
server_key_version: "encryptionkey@0"
I20260926 02:53:44.909562 28870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:53:44.913297 28885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:44.914647 28870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:53:44.914842 28870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "c783db2d5df44d6a95fe9a73d04791c8"
format_stamp: "Formatted at 2026-09-26 02:53:44 on dist-test-slave-ktng"
server_key: "95d3eb240da1ae7e6ff87196fc93f23a"
server_key_iv: "d1bd01e3bf9e353918a941590fb0c78a"
server_key_version: "encryptionkey@0"
I20260926 02:53:44.915066 28870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:44.930048 28870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:44.934361 28870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:44.934654 28870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:44.935587 28870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:44.936870 28870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:44.936981 28870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:44.937122 28870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:44.937199 28870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:44.971611 28870 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:36509
I20260926 02:53:44.971652 28997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:36509 every 8 connection(s)
I20260926 02:53:44.972890 28870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:53:44.979390 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 28870
I20260926 02:53:44.979597 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:53:44.980067 20199 external_mini_cluster.cc:1428] Setting key bff9c10e278b845445d25bbcd6b9d810
Sep 26 02:53:44 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391224, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:53:44.993778 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:44.975557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:51369 (local address 127.19.185.254:39061)
0926 02:53:44.975739 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:44.975744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:44.976947 (+ 1203us) server_negotiation.cc:407] Connection header received
0926 02:53:44.978830 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:44.978835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:44.978927 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:44.979040 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:44.980849 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:44.981669 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:44.983241 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:44.983716 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:44.983859 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:44.987530 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:44.987535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:44.987539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:44.987577 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:44.990641 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:44.991412 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:44.991417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:44.991421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:44.991552 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:44.992005 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:44.992010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:44.992013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:44.992278 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:44.992432 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:44.993172 (+ 740us) server_negotiation.cc:299] Negotiation successful
0926 02:53:44.993390 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:53:44.995008 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:44.976060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:51369)
0926 02:53:44.976790 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:44.976830 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:44.978499 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:44.979409 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:44.979418 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:44.980038 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:44.980664 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:44.980682 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:44.981858 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:44.981864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:44.983083 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:44.983094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:44.983948 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:44.983956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:44.984217 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:44.984912 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:44.984934 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:44.987239 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:53:44.990785 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:44.990794 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:44.990813 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:44.991241 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:44.991667 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:53:44.991672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:53:44.991676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:53:44.991878 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:53:44.992418 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:53:44.992427 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:53:44.992780 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 02:53:44.993077 (+ 297us) client_negotiation.cc:240] Negotiation successful
0926 02:53:44.993624 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":310,"threads_started":1}
I20260926 02:53:44.996584 28998 heartbeater.cc:347] Connected to a master server at 127.19.185.254:39061
I20260926 02:53:44.996922 28998 heartbeater.cc:464] Registering TS with master...
I20260926 02:53:44.997694 28998 heartbeater.cc:511] Master 127.19.185.254:39061 requested a full tablet report, sending...
I20260926 02:53:44.998145 28731 heartbeater.cc:503] Master 127.19.185.254:39061 was elected leader, sending a full tablet report...
I20260926 02:53:44.999274 28541 ts_manager.cc:194] Registered new tserver with Master: c783db2d5df44d6a95fe9a73d04791c8 (127.19.185.195:36509)
I20260926 02:53:45.000257 28541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:51369
I20260926 02:53:45.008277 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.193.keytab.
Entry for principal oryx/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.193.keytab.
WARNING: no policy specified for oryx/127.19.185.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.185.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.194.keytab.
Entry for principal oryx/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.194.keytab.
WARNING: no policy specified for oryx/127.19.185.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.195.keytab.
Entry for principal oryx/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.195.keytab.
I20260926 02:53:45.151423 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 28603
Sep 26 02:53:45 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:53:45.170857 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:45.152663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.19.185.193:46339)
0926 02:53:45.153445 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:45.153453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:45.153485 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:53:45.153546 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:45.153552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:45.153777 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:45.153909 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:45.155243 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:45.156430 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:45.157725 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:45.158393 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:45.158504 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:45.162778 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:45.162796 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:45.162816 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:45.162869 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:45.167008 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:45.167844 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:45.167853 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:45.167857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:45.168046 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:45.168474 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:45.168479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:45.168482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:45.168945 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:45.169230 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:45.170200 (+ 970us) server_negotiation.cc:299] Negotiation successful
0926 02:53:45.170446 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":438,"threads_started":1}
I20260926 02:53:45.172978 28726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:45.256471 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 28603
I20260926 02:53:45.273953 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 28737
Sep 26 02:53:45 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:53:45.291217 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:45.274900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50134 (local address 127.19.185.194:36579)
0926 02:53:45.275557 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:45.275564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:45.275590 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:53:45.275649 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:45.275655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:45.275892 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:45.276150 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:45.277770 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:45.278826 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:45.280277 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:45.280807 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:45.280909 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:45.285301 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:45.285312 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:45.285326 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:45.285366 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:45.288378 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:45.289102 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:45.289110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:45.289112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:45.289240 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:45.289638 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:45.289645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:45.289648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:45.289981 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:45.290292 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:45.290600 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 02:53:45.290861 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":414,"threads_started":1}
I20260926 02:53:45.292544 28860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:45.374106 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 28737
I20260926 02:53:45.390264 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 28870
Sep 26 02:53:45 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:53:45.406869 29012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:45.391507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.19.185.195:36509)
0926 02:53:45.392122 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:45.392129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:45.392157 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:53:45.392216 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:45.392221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:45.392428 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:45.392572 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:45.393825 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:45.394877 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:45.396248 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:45.396914 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:45.397031 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:45.401365 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:45.401376 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:45.401391 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:45.401423 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:45.404396 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:45.405006 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:45.405014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:45.405016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:45.405188 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:45.405509 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:45.405512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:45.405514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:45.405837 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:45.406070 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:45.406290 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 02:53:45.406528 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":315,"threads_started":1}
I20260926 02:53:45.408212 28993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:45.493549 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 28870
I20260926 02:53:45.509537 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 28510
I20260926 02:53:45.510834 28572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:53:45.613843 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 28510
I20260926 02:53:45.634544 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:39061
--webserver_interface=127.19.185.254
--webserver_port=36507
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:39061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:53:45.959620 29015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:45.959913 29015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:45.960008 29015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:45.967033 29015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:53:45.967132 29015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:45.967177 29015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:45.967234 29015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:53:45.967278 29015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:53:45.977087 29015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:39061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:39061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=36507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:45.978385 29015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:45.979698 29015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:45.992909 29023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:45.992995 29020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:45.992957 29021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:45.999375 29015 server_base.cc:1034] running on GCE node
I20260926 02:53:46.000241 29015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:46.001780 29015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:46.003001 29015 hybrid_clock.cc:648] HybridClock initialized: now 1790391226002949 us; error 37 us; skew 500 ppm
I20260926 02:53:46.003432 29015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:45 dist-test-slave-ktng krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391225, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:46.006716 29015 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:53:46.008391 29015 webserver.cc:460] Webserver started at http://127.19.185.254:36507/ using document root <none> and password file <none>
I20260926 02:53:46.009080 29015 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:46.009200 29015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:46.014714 29015 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:53:46.017500 29030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:46.018899 29015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:53:46.019068 29015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "98534291b94942f8b86d68e94534e563"
format_stamp: "Formatted at 2026-09-26 02:53:43 on dist-test-slave-ktng"
server_key: "14bf92af1d4b618af618d604f766ec50"
server_key_iv: "96414a5a612fbff27221e66ceaf40816"
server_key_version: "encryptionkey@0"
I20260926 02:53:46.020074 29015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:46.041999 29015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:46.046676 29015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:46.046980 29015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:46.066864 29015 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:39061
I20260926 02:53:46.066882 29081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:39061 every 8 connection(s)
I20260926 02:53:46.068492 29015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:53:46.075361 29082 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:53:46.076397 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 29015
I20260926 02:53:46.077378 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:46339
--local_ip_for_outbound_sockets=127.19.185.193
--tserver_master_addrs=127.19.185.254:39061
--webserver_port=40359
--webserver_interface=127.19.185.193
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.185.193
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:53:46.080518 29082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563: Bootstrap starting.
I20260926 02:53:46.101444 29082 log.cc:824] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563: Log is configured to *not* fsync() on all Append() calls
I20260926 02:53:46.110545 29082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:53:46.111102 29082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563: Bootstrap complete.
I20260926 02:53:46.120497 29082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98534291b94942f8b86d68e94534e563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39061 } }
I20260926 02:53:46.121372 29082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98534291b94942f8b86d68e94534e563, State: Initialized, Role: FOLLOWER
I20260926 02:53:46.122066 29082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98534291b94942f8b86d68e94534e563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39061 } }
I20260926 02:53:46.122314 29082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:53:46.122450 29082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:53:46.122627 29082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:53:46.126858 29082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98534291b94942f8b86d68e94534e563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39061 } }
I20260926 02:53:46.127398 29082 leader_election.cc:302] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98534291b94942f8b86d68e94534e563; no voters:
I20260926 02:53:46.128054 29082 leader_election.cc:288] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:53:46.128211 29086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:53:46.129614 29086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [term 2 LEADER]: Becoming Leader. State: Replica: 98534291b94942f8b86d68e94534e563, State: Running, Role: LEADER
I20260926 02:53:46.130162 29086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98534291b94942f8b86d68e94534e563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39061 } }
I20260926 02:53:46.130403 29082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:53:46.132571 29088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98534291b94942f8b86d68e94534e563. Latest consensus state: current_term: 2 leader_uuid: "98534291b94942f8b86d68e94534e563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98534291b94942f8b86d68e94534e563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39061 } } }
I20260926 02:53:46.132892 29088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:46.133420 29087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "98534291b94942f8b86d68e94534e563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98534291b94942f8b86d68e94534e563" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39061 } } }
I20260926 02:53:46.133742 29087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563 [sys.catalog]: This master's current role is: LEADER
I20260926 02:53:46.133930 29093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:53:46.148689 29093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:53:46.150075 29093 catalog_manager.cc:1259] Loaded cluster ID: 94376fc2b3e147bfa80175c67a1aeebf
I20260926 02:53:46.150190 29093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:53:46.154196 29093 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:53:46.155678 29093 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98534291b94942f8b86d68e94534e563: Loaded TSK: 0
I20260926 02:53:46.157382 29093 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:53:46.428669 29084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:46.429003 29084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:46.429171 29084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:46.436515 29084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:46.436664 29084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:46.436843 29084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:53:46.446619 29084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.193.keytab
--principal=oryx/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:46339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=40359
--tserver_master_addrs=127.19.185.254:39061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:46.448486 29084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:46.450090 29084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:46.464877 29109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:46.464876 29111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:46.465715 29108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:46.466074 29084 server_base.cc:1034] running on GCE node
I20260926 02:53:46.466691 29084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:46.467787 29084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:46.469022 29084 hybrid_clock.cc:648] HybridClock initialized: now 1790391226468951 us; error 58 us; skew 500 ppm
I20260926 02:53:46.469341 29084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:46 dist-test-slave-ktng krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391226, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:46.472615 29084 init.cc:377] Logged in from keytab as oryx/127.19.185.193@KRBTEST.COM (short username oryx)
I20260926 02:53:46.474181 29084 webserver.cc:460] Webserver started at http://127.19.185.193:40359/ using document root <none> and password file <none>
I20260926 02:53:46.474871 29084 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:46.474982 29084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:46.480418 29084 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:53:46.483232 29118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:46.484751 29084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:53:46.484959 29084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "48b6b24a3c4a4c15a9613df10b9e12c0"
format_stamp: "Formatted at 2026-09-26 02:53:43 on dist-test-slave-ktng"
server_key: "9b1064dedee7bcc3263f56cc349de996"
server_key_iv: "370fc3ad3ff0f604021000c2f6dda223"
server_key_version: "encryptionkey@0"
I20260926 02:53:46.485935 29084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:46.502928 29084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:46.507412 29084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:46.507699 29084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:46.508716 29084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:46.510032 29084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:46.510162 29084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:46.510298 29084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:46.510392 29084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:46.545923 29084 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:46339
I20260926 02:53:46.546036 29230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:46339 every 8 connection(s)
I20260926 02:53:46.547487 29084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:53:46.553900 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 29084
I20260926 02:53:46.554517 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:36579
--local_ip_for_outbound_sockets=127.19.185.194
--tserver_master_addrs=127.19.185.254:39061
--webserver_port=35131
--webserver_interface=127.19.185.194
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.185.194
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:53:46 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:46 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:46.567678 29234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:46.550730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:48709)
0926 02:53:46.551626 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:46.551680 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:46.553491 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:46.555067 (+ 1576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:46.555080 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:46.555899 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:46.556711 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:46.556740 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:46.558354 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:46.558362 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:46.560524 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:46.560543 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:46.561683 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:46.561688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:46.561879 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:46.562618 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:46.562640 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:46.566575 (+ 3935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":337,"threads_started":1}
W20260926 02:53:46.568216 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:46.550650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:48709 (local address 127.19.185.254:39061)
0926 02:53:46.551503 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:46.551520 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:46.551810 (+ 290us) server_negotiation.cc:407] Connection header received
0926 02:53:46.554114 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:46.554134 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:46.554457 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:46.554830 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:46.556915 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:46.558181 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:46.560698 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:46.561462 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:46.561681 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:46.567013 (+ 5332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:48709: BlockingRecv error: recv got EOF from 127.19.185.193:48709 (error 108)
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":352,"threads_started":1}
W20260926 02:53:46.568521 29231 heartbeater.cc:644] Failed to heartbeat to 127.19.185.254:39061 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.185.254:39061: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Sep 26 02:53:46 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:46 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:46.578205 29234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:46.569148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:39781)
0926 02:53:46.569358 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:46.569379 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:46.569490 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:46.569918 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:46.569921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:46.570291 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:46.570428 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:46.570435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:46.572077 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:46.572084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:46.573007 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:46.573019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:46.574184 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:46.574189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:46.574308 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:46.574999 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:46.575027 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:46.577997 (+ 2970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 02:53:46.578548 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:46.569402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:39781 (local address 127.19.185.254:39061)
0926 02:53:46.569605 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:46.569612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:46.569640 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:53:46.569697 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:46.569704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:46.569785 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:46.569892 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:46.570571 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:46.571897 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:46.573236 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:46.574021 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:46.574172 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:46.578271 (+ 4099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:39781: BlockingRecv error: recv got EOF from 127.19.185.193:39781 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 02:53:46 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:46 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:46.586910 29234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:46.578941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:42171)
0926 02:53:46.579158 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:46.579172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:46.579244 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:46.579689 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:46.579692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:46.579937 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:46.580065 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:46.580072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:46.581474 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:46.581478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:46.582380 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:46.582391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:46.583275 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:46.583279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:46.583465 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:46.584028 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:46.584044 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:46.586713 (+ 2669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118}
W20260926 02:53:46.587224 29231 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:53:46.587205 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:46.579166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:42171 (local address 127.19.185.254:39061)
0926 02:53:46.579369 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:46.579374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:46.579394 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:53:46.579450 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:46.579458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:46.579546 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:46.579676 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:46.580196 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:46.581316 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:46.582501 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:46.583143 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:46.583261 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:46.586956 (+ 3695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:42171: BlockingRecv error: recv got EOF from 127.19.185.193:42171 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 02:53:46.888453 29235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:46.888756 29235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:46.888849 29235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:46.896416 29235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:46.896526 29235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:46.896638 29235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:53:46.906416 29235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.194.keytab
--principal=oryx/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:36579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=35131
--tserver_master_addrs=127.19.185.254:39061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:46.907773 29235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:46.909009 29235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:46.922456 29241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:46.922499 29240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:46.923235 29235 server_base.cc:1034] running on GCE node
W20260926 02:53:46.923247 29243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:46.924082 29235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:46.925262 29235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:46.926503 29235 hybrid_clock.cc:648] HybridClock initialized: now 1790391226926461 us; error 44 us; skew 500 ppm
I20260926 02:53:46.926836 29235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:46 dist-test-slave-ktng krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391226, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:46.930039 29235 init.cc:377] Logged in from keytab as oryx/127.19.185.194@KRBTEST.COM (short username oryx)
I20260926 02:53:46.931691 29235 webserver.cc:460] Webserver started at http://127.19.185.194:35131/ using document root <none> and password file <none>
I20260926 02:53:46.932431 29235 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:46.932603 29235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:46.937847 29235 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 02:53:46.940582 29250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:46.941929 29235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:53:46.942168 29235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "bb6e968404f64955b4f709f7c21bf089"
format_stamp: "Formatted at 2026-09-26 02:53:44 on dist-test-slave-ktng"
server_key: "b19beca6a0f4718e34d013add9c9737c"
server_key_iv: "d39bd18bae4cf237aad95699f8e60086"
server_key_version: "encryptionkey@0"
I20260926 02:53:46.943156 29235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:46.973390 29235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:46.977655 29235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:46.977962 29235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:46.978981 29235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:46.980433 29235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:46.980561 29235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:46.980703 29235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:46.980798 29235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:47.015793 29235 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:36579
I20260926 02:53:47.015838 29362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:36579 every 8 connection(s)
I20260926 02:53:47.017238 29235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:53:47.020479 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 29235
I20260926 02:53:47.021090 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:36509
--local_ip_for_outbound_sockets=127.19.185.195
--tserver_master_addrs=127.19.185.254:39061
--webserver_port=37129
--webserver_interface=127.19.185.195
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.185.195
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:53:47.035230 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:47.019802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:52267 (local address 127.19.185.254:39061)
0926 02:53:47.020011 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:47.020016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:47.021422 (+ 1406us) server_negotiation.cc:407] Connection header received
0926 02:53:47.023496 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:47.023501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:47.023596 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:47.023711 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:47.025396 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.026110 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.028607 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.029030 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.029153 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.034975 (+ 5822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:52267: BlockingRecv error: recv got EOF from 127.19.185.194:52267 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 02:53:47.035995 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:47.020391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:52267)
0926 02:53:47.021249 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:47.021305 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:47.023119 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:47.023969 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:47.023985 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:47.024569 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:47.025222 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:47.025243 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.026288 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.026298 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:47.028374 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:47.028393 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.029275 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.029284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:47.029615 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.030524 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:47.030559 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:47.034761 (+ 4202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":389,"threads_started":1}
W20260926 02:53:47.036684 29363 heartbeater.cc:644] Failed to heartbeat to 127.19.185.254:39061 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.185.254:39061: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:47.047081 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:47.037309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:35353)
0926 02:53:47.037600 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:47.037635 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:47.037768 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:47.038110 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:47.038117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:47.038585 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:47.038821 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:47.038834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.039892 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.039899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:47.041639 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:47.041658 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.042348 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.042358 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:47.042591 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.043248 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:47.043279 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:47.046819 (+ 3540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151}
W20260926 02:53:47.047577 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:47.037400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:35353 (local address 127.19.185.254:39061)
0926 02:53:47.037557 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:47.037562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:47.037711 (+ 149us) server_negotiation.cc:407] Connection header received
0926 02:53:47.037881 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:47.037884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:47.037958 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:47.038040 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:47.038969 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.039746 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.041797 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.042184 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.042295 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.047082 (+ 4787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:35353: BlockingRecv error: recv got EOF from 127.19.185.194:35353 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:47.056265 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:47.047902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:57867)
0926 02:53:47.048081 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:47.048100 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:47.048201 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:47.048532 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:47.048535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:47.048793 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:47.048932 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:47.048939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.050392 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.050398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:47.051577 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:47.051589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.052598 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.052602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:47.052734 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.053339 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:47.053356 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:47.056021 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 02:53:47.056491 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:47.047976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:57867 (local address 127.19.185.254:39061)
0926 02:53:47.048130 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:47.048136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:47.048154 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:53:47.048325 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:47.048330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:47.048402 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:47.048501 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:47.049123 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.050213 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.051703 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.052436 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.052567 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.056261 (+ 3694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:57867: BlockingRecv error: recv got EOF from 127.19.185.194:57867 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 02:53:47.056591 29363 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:53:47.362087 29366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:53:47.362380 29366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:53:47.362473 29366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:53:47.369890 29366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:53:47.370010 29366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:53:47.370134 29366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:53:47.380245 29366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/krb5kdc/oryx_127.19.185.195.keytab
--principal=oryx/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:36509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=37129
--tserver_master_addrs=127.19.185.254:39061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:53:47.381548 29366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:53:47.382807 29366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:53:47.397094 29372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:47.397131 29371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:53:47.397182 29374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:53:47.397667 29366 server_base.cc:1034] running on GCE node
I20260926 02:53:47.398343 29366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:53:47.399375 29366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:53:47.400638 29366 hybrid_clock.cc:648] HybridClock initialized: now 1790391227400594 us; error 34 us; skew 500 ppm
I20260926 02:53:47.400980 29366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391227, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:53:47.404386 29366 init.cc:377] Logged in from keytab as oryx/127.19.185.195@KRBTEST.COM (short username oryx)
I20260926 02:53:47.405846 29366 webserver.cc:460] Webserver started at http://127.19.185.195:37129/ using document root <none> and password file <none>
I20260926 02:53:47.406574 29366 fs_manager.cc:359] Metadata directory not provided
I20260926 02:53:47.406733 29366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:53:47.412328 29366 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:53:47.415115 29381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:47.416667 29366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:53:47.416865 29366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "c783db2d5df44d6a95fe9a73d04791c8"
format_stamp: "Formatted at 2026-09-26 02:53:44 on dist-test-slave-ktng"
server_key: "95d3eb240da1ae7e6ff87196fc93f23a"
server_key_iv: "d1bd01e3bf9e353918a941590fb0c78a"
server_key_version: "encryptionkey@0"
I20260926 02:53:47.417809 29366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:53:47.453058 29366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:53:47.457816 29366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:53:47.458146 29366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:53:47.459172 29366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:53:47.460796 29366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:53:47.460922 29366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:47.461066 29366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:53:47.461154 29366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:53:47.497414 29366 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:36509
I20260926 02:53:47.497532 29493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:36509 every 8 connection(s)
I20260926 02:53:47.498983 29366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:53:47.506587 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 29366
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:53:47.517165 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:47.501650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:41381 (local address 127.19.185.254:39061)
0926 02:53:47.501842 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:47.501847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:47.503066 (+ 1219us) server_negotiation.cc:407] Connection header received
0926 02:53:47.505008 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:47.505014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:47.505102 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:47.505226 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:47.506737 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.507587 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.510033 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.510523 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.510670 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.516929 (+ 6259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:41381: BlockingRecv error: recv got EOF from 127.19.185.195:41381 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:53:47.517973 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:47.502168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:41381)
0926 02:53:47.502905 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:47.502942 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:47.504745 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:47.505467 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:47.505475 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:47.505987 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:47.506555 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:47.506572 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.507761 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.507768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:47.509809 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:47.509827 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.510739 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.510744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:47.510954 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.511696 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:47.511728 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:47.516723 (+ 4995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":294,"threads_started":1}
W20260926 02:53:47.518599 29494 heartbeater.cc:644] Failed to heartbeat to 127.19.185.254:39061 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.185.254:39061: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
I20260926 02:53:47.525045 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:47.507670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.19.185.254:39061)
0926 02:53:47.508423 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:47.508427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:47.508442 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:53:47.508482 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:47.508485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:47.508537 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:47.508622 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:47.510539 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.511563 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.513443 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.514012 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.514110 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.517579 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:53:47.517587 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:53:47.517603 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:53:47.517651 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:53:47.521389 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:47.522216 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:47.522226 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:47.522234 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:47.522502 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:53:47.522882 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:53:47.522888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:53:47.522891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:53:47.523427 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:53:47.523786 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0926 02:53:47.524345 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 02:53:47.524680 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":550,"threads_started":1}
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:47.527621 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:47.519243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:49967)
0926 02:53:47.519430 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:47.519448 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:47.519589 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:47.519922 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:47.519928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:47.520330 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:47.520504 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:47.520514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.521694 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.521700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:47.522655 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:47.522666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.523401 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.523407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:47.523652 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.524292 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:47.524313 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:47.527338 (+ 3025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 02:53:47.527787 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:47.519377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:49967 (local address 127.19.185.254:39061)
0926 02:53:47.519579 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:47.519588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:47.519608 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:53:47.519709 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:47.519714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:47.519784 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:47.519859 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:47.520645 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.521555 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.522796 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.523229 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.523390 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.527543 (+ 4153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:49967: BlockingRecv error: recv got EOF from 127.19.185.195:49967 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:47.535826 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:47.528264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:57747)
0926 02:53:47.528431 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:47.528450 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:47.528535 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:47.529076 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:47.529084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:47.529378 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:47.529516 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:47.529523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.530497 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.530501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:47.531721 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:47.531732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.532433 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.532437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:47.532558 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.533096 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:47.533112 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:47.535597 (+ 2485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 02:53:47.536000 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:47.528433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:57747 (local address 127.19.185.254:39061)
0926 02:53:47.528686 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:47.528693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:47.528711 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:53:47.528758 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:47.528762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:47.528825 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:47.528926 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:47.529645 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.530355 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.531891 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.532301 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.532394 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.535798 (+ 3404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:57747: BlockingRecv error: recv got EOF from 127.19.185.195:57747 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
W20260926 02:53:47.536166 29494 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:47 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:47.596975 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:47.587905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:53337)
0926 02:53:47.588447 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:47.588470 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:47.588618 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:47.589094 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:47.589099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:47.589743 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:47.589929 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:47.589939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.591137 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.591141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:47.592192 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:47.592204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.592913 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.592918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:47.593120 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.593679 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:47.593696 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:47.596754 (+ 3058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":266,"threads_started":1}
W20260926 02:53:47.597378 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:47.587960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:53337 (local address 127.19.185.254:39061)
0926 02:53:47.588153 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:47.588159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:47.588548 (+ 389us) server_negotiation.cc:407] Connection header received
0926 02:53:47.588759 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:47.588765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:47.588833 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:47.588961 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:47.590095 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.591002 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.592334 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:47.592745 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:47.592837 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:47.596993 (+ 4156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:53337: BlockingRecv error: recv got EOF from 127.19.185.193:53337 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 02:53:48 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:48 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:48.066532 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:48.057261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:45283)
0926 02:53:48.057802 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:48.057819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:48.057936 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:48.058332 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:48.058337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:48.058996 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:48.059237 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:48.059247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:48.060412 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:48.060417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:48.061434 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:48.061445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:48.062188 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:48.062194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:48.062368 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:48.063189 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:48.063208 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:48.066187 (+ 2979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":300,"threads_started":1}
W20260926 02:53:48.066681 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:48.057335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:45283 (local address 127.19.185.254:39061)
0926 02:53:48.057526 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:48.057531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:48.057904 (+ 373us) server_negotiation.cc:407] Connection header received
0926 02:53:48.058066 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:48.058070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:48.058160 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:48.058297 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:48.059421 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:48.060243 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:48.061584 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:48.062002 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:48.062161 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:48.066410 (+ 4249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:45283: BlockingRecv error: recv got EOF from 127.19.185.194:45283 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 02:53:48 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:48 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:48.544881 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:48.536735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:42615)
0926 02:53:48.537232 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:48.537248 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:48.537358 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:48.537661 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:48.537664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:48.538127 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:48.538325 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:48.538333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:48.539315 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:48.539319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:48.540265 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:48.540277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:48.540917 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:48.540922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:48.541060 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:48.541614 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:48.541639 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:48.544566 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":231,"threads_started":1}
W20260926 02:53:48.545064 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:48.536997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:42615 (local address 127.19.185.254:39061)
0926 02:53:48.537226 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:48.537231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:48.537322 (+ 91us) server_negotiation.cc:407] Connection header received
0926 02:53:48.537454 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:48.537458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:48.537518 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:48.537596 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:48.538448 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:48.539169 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:48.540385 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:48.540769 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:48.540859 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:48.544792 (+ 3933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:42615: BlockingRecv error: recv got EOF from 127.19.185.195:42615 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 26 02:53:48 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:48 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:48.606709 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:48.597961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:44495)
0926 02:53:48.598450 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:48.598466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:48.598627 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:48.599021 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:48.599026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:48.599479 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:48.599642 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:48.599659 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:48.600710 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:48.600715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:48.601594 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:48.601604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:48.602341 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:48.602346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:48.602510 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:48.603174 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:48.603201 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:48.606355 (+ 3154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":269,"threads_started":1}
W20260926 02:53:48.606786 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:48.598038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:44495 (local address 127.19.185.254:39061)
0926 02:53:48.598316 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:48.598323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:48.598546 (+ 223us) server_negotiation.cc:407] Connection header received
0926 02:53:48.598791 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:48.598797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:48.598884 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:48.598991 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:48.599788 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:48.600563 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:48.601754 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:48.602175 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:48.602329 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:48.606537 (+ 4208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:44495: BlockingRecv error: recv got EOF from 127.19.185.193:44495 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 02:53:49 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:49 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:49.076851 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:49.067643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:57429)
0926 02:53:49.068209 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:49.068224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:49.068344 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:49.068697 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:49.068700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:49.069068 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:49.069237 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:49.069244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:49.070606 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:49.070610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:49.071605 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:49.071617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:49.072570 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:49.072574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:49.072700 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:49.073328 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:49.073345 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:49.076626 (+ 3281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":331,"threads_started":1}
W20260926 02:53:49.077049 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:49.067742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:57429 (local address 127.19.185.254:39061)
0926 02:53:49.067929 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:49.067935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:49.068303 (+ 368us) server_negotiation.cc:407] Connection header received
0926 02:53:49.068460 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:49.068465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:49.068541 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:49.068644 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:49.069362 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:49.070462 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:49.071747 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:49.072419 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:49.072548 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:49.076815 (+ 4267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:57429: BlockingRecv error: recv got EOF from 127.19.185.194:57429 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 02:53:49 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:49 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:49.555171 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:49.546061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:41833)
0926 02:53:49.546661 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:49.546680 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:49.546817 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:49.547220 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:49.547225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:49.547757 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:49.547958 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:49.547969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:49.549231 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:49.549237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:49.550501 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:49.550519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:49.551519 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:49.551524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:49.551634 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:49.552291 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:49.552312 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:49.554941 (+ 2629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":313,"threads_started":1}
W20260926 02:53:49.555438 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:49.546298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:41833 (local address 127.19.185.254:39061)
0926 02:53:49.546505 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:49.546511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:49.546830 (+ 319us) server_negotiation.cc:407] Connection header received
0926 02:53:49.546919 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:49.546923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:49.546990 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:49.547167 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:49.548118 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:49.548998 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:49.550737 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:49.551385 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:49.551481 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:49.555162 (+ 3681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:41833: BlockingRecv error: recv got EOF from 127.19.185.195:41833 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 02:53:49 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:49 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:49.616232 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:49.607751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:55135)
0926 02:53:49.608339 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:49.608355 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:49.608519 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:49.608915 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:49.608921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:49.609462 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:49.609630 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:49.609640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:49.610855 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:49.610860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:49.611872 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:49.611885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:49.612531 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:49.612535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:49.612674 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:49.613285 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:49.613302 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:49.615972 (+ 2670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":319,"threads_started":1}
W20260926 02:53:49.616380 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:49.607908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:55135 (local address 127.19.185.254:39061)
0926 02:53:49.608179 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:49.608187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:49.608453 (+ 266us) server_negotiation.cc:407] Connection header received
0926 02:53:49.608664 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:49.608670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:49.608753 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:49.608870 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:49.609785 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:49.610674 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:49.612014 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:49.612400 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:49.612492 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:49.616148 (+ 3656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:55135: BlockingRecv error: recv got EOF from 127.19.185.193:55135 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 26 02:53:50 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:50 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:50.087514 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:50.077892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:36703)
0926 02:53:50.078488 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:50.078516 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:50.078688 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:50.079072 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:50.079076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:50.079534 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:50.079711 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:50.079721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:50.080821 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:50.080831 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:50.081807 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:50.081819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:50.082527 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:50.082532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:50.082684 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:50.083685 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:50.083712 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:50.087122 (+ 3410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":346,"threads_started":1}
W20260926 02:53:50.087776 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:50.077971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:36703 (local address 127.19.185.254:39061)
0926 02:53:50.078180 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:50.078184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:50.078626 (+ 442us) server_negotiation.cc:407] Connection header received
0926 02:53:50.078799 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:50.078803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:50.078863 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:50.078948 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:50.079875 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:50.080685 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:50.081949 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:50.082358 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:50.082453 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:50.087470 (+ 5017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:36703: BlockingRecv error: recv got EOF from 127.19.185.194:36703 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 02:53:50 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:50 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:50.564954 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:50.556372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:50491)
0926 02:53:50.556938 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:50.556955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:50.557072 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:50.557374 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:50.557377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:50.557730 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:50.557888 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:50.557897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:50.559012 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:50.559016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:50.559970 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:50.559983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:50.560934 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:50.560939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:50.561157 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:50.561864 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:50.561881 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:50.564688 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":307,"threads_started":1}
W20260926 02:53:50.565037 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:50.556365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:50491 (local address 127.19.185.254:39061)
0926 02:53:50.556659 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:50.556670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:50.557070 (+ 400us) server_negotiation.cc:407] Connection header received
0926 02:53:50.557178 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:50.557182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:50.557245 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:50.557332 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:50.558074 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:50.558855 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:50.560172 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:50.560642 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:50.560754 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:50.564872 (+ 4118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:50491: BlockingRecv error: recv got EOF from 127.19.185.195:50491 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 26 02:53:50 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:50 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:50.626101 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:50.617161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:38101)
0926 02:53:50.617675 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:50.617690 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:50.617861 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:50.618225 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:50.618229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:50.618616 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:50.618766 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:50.618775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:50.619872 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:50.619879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:50.620917 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:50.620929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:50.621710 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:50.621716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:50.621893 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:50.622572 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:50.622589 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:50.625885 (+ 3296us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":283,"threads_started":1}
W20260926 02:53:50.626396 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:50.617225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:38101 (local address 127.19.185.254:39061)
0926 02:53:50.617503 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:50.617511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:50.617781 (+ 270us) server_negotiation.cc:407] Connection header received
0926 02:53:50.618024 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:50.618029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:50.618097 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:50.618202 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:50.618914 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:50.619744 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:50.621057 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:50.621549 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:50.621697 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:50.626069 (+ 4372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:38101: BlockingRecv error: recv got EOF from 127.19.185.193:38101 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 26 02:53:51 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:51 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:51.097469 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:51.088603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:41443)
0926 02:53:51.089155 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:51.089175 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:51.089309 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:51.089735 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:51.089738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:51.090182 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:51.090394 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:51.090403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:51.091552 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:51.091556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:51.092550 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:51.092561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:51.093250 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:51.093254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:51.093370 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:51.093979 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:51.093996 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:51.097059 (+ 3063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":313,"threads_started":1}
W20260926 02:53:51.097508 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:51.088706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:41443 (local address 127.19.185.254:39061)
0926 02:53:51.088886 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:51.088893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:51.089260 (+ 367us) server_negotiation.cc:407] Connection header received
0926 02:53:51.089465 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:51.089470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:51.089586 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:51.089695 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:51.090522 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:51.091440 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:51.092681 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:51.093098 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:51.093195 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:51.097331 (+ 4136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:41443: BlockingRecv error: recv got EOF from 127.19.185.194:41443 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 02:53:51 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:51 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:51.574867 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:51.566014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:45421)
0926 02:53:51.566549 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:51.566566 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:51.566730 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:51.567068 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:51.567073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:51.567482 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:51.567629 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:51.567638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:51.568686 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:51.568691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:51.570077 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:51.570094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:51.570931 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:51.570936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:51.571065 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:51.571671 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:51.571687 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:51.574529 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":305,"threads_started":1}
W20260926 02:53:51.574972 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:51.566090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:45421 (local address 127.19.185.254:39061)
0926 02:53:51.566322 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:51.566330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:51.566661 (+ 331us) server_negotiation.cc:407] Connection header received
0926 02:53:51.566856 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:51.566861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:51.566933 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:51.567037 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:51.567770 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:51.568539 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:51.570215 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:51.570780 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:51.570902 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:51.574729 (+ 3827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:45421: BlockingRecv error: recv got EOF from 127.19.185.195:45421 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 26 02:53:51 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:51 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:51.635653 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:51.627107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:48551)
0926 02:53:51.627731 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:51.627746 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:51.627904 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:51.628244 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:51.628248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:51.628672 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:51.628823 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:51.628832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:51.630035 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:51.630058 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:51.631091 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:51.631102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:51.631844 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:51.631849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:51.631967 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:51.632639 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:51.632657 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:51.635411 (+ 2754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":359,"threads_started":1}
W20260926 02:53:51.635819 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:51.627196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:48551 (local address 127.19.185.254:39061)
0926 02:53:51.627552 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:51.627560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:51.627835 (+ 275us) server_negotiation.cc:407] Connection header received
0926 02:53:51.628037 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:51.628043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:51.628109 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:51.628218 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:51.628982 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:51.629886 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:51.631266 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:51.631695 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:51.631795 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:51.635593 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:48551: BlockingRecv error: recv got EOF from 127.19.185.193:48551 (error 108)
Metrics: {"server-negotiator.queue_time_us":182}
Sep 26 02:53:52 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:52 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:52.106963 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:52.098555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:52051)
0926 02:53:52.099262 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:52.099278 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:52.099448 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:52.099795 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:52.099800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:52.100195 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:52.100355 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:52.100364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:52.101361 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:52.101369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:52.102291 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:52.102302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:52.103028 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:52.103033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:52.103211 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:52.103848 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:52.103866 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:52.106613 (+ 2747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":424,"threads_started":1}
W20260926 02:53:52.107074 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:52.098775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:52051 (local address 127.19.185.254:39061)
0926 02:53:52.098963 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:52.098971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:52.099389 (+ 418us) server_negotiation.cc:407] Connection header received
0926 02:53:52.099567 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:52.099572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:52.099664 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:52.099765 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:52.100497 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:52.101224 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:52.102431 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:52.102887 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:52.103018 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:52.106814 (+ 3796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:52051: BlockingRecv error: recv got EOF from 127.19.185.194:52051 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 02:53:52 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:52 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:52.583967 29513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:52.575995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:34521)
0926 02:53:52.576560 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:52.576576 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:52.576704 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:52.576981 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:52.576984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:52.577350 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:52.577504 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:52.577512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:52.578538 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:52.578541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:52.579536 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:52.579547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:52.580341 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:52.580345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:52.580458 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:52.581016 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:52.581033 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:52.583729 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":322,"threads_started":1}
W20260926 02:53:52.584177 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:52.576083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:34521 (local address 127.19.185.254:39061)
0926 02:53:52.576251 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:52.576255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:52.576657 (+ 402us) server_negotiation.cc:407] Connection header received
0926 02:53:52.576806 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:52.576809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:52.576870 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:52.576941 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:52.577638 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:52.578416 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:52.579676 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:52.580197 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:52.580319 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:52.583917 (+ 3598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:34521: BlockingRecv error: recv got EOF from 127.19.185.195:34521 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 02:53:52 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:52 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:52.644765 29514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:52.636643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:45145)
0926 02:53:52.637207 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:52.637223 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:52.637375 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:52.637739 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:52.637745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:52.638158 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:52.638309 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:52.638318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:52.639338 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:52.639344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:52.640233 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:52.640244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:52.640953 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:52.640959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:52.641139 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:52.641744 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:52.641764 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:52.644439 (+ 2675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":310,"threads_started":1}
W20260926 02:53:52.644872 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:52.636792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:45145 (local address 127.19.185.254:39061)
0926 02:53:52.637025 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:52.637032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:52.637309 (+ 277us) server_negotiation.cc:407] Connection header received
0926 02:53:52.637499 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:52.637504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:52.637592 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:52.637696 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:52.638449 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:52.639183 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:52.640373 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:52.640798 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:52.640938 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:52.644630 (+ 3692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:45145: BlockingRecv error: recv got EOF from 127.19.185.193:45145 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 02:53:53 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:53 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:53.116037 29515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:53.108057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:54543)
0926 02:53:53.108659 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:53.108674 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:53.108786 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:53.109067 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:53.109072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:53.109445 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:53.109605 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:53.109613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:53.110589 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:53.110593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:53.111520 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:53.111532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:53.112180 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:53.112184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:53.112303 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:53.112869 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:53.112896 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:53.115719 (+ 2823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":341,"threads_started":1}
W20260926 02:53:53.116215 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:53.108119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:54543 (local address 127.19.185.254:39061)
0926 02:53:53.108275 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:53.108280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:53.108752 (+ 472us) server_negotiation.cc:407] Connection header received
0926 02:53:53.108879 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:53.108883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:53.108945 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:53.109025 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:53.109737 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:53.110449 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:53.111655 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:53.112051 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:53.112145 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:53.115974 (+ 3829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:54543: BlockingRecv error: recv got EOF from 127.19.185.194:54543 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 02:53:53 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:53 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:53.599668 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:53.584949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:39125)
0926 02:53:53.585575 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:53.585591 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:53.585748 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:53.586187 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:53.586194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:53.586599 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:53.586757 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:53.586766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:53.587919 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:53.587926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:53.589237 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:53.589251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:53.590364 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:53.590370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:53.590577 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:53.596510 (+ 5933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:53.596529 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:53.599410 (+ 2881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":348,"threads_started":1}
W20260926 02:53:53.599905 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:53.585204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:39125 (local address 127.19.185.254:39061)
0926 02:53:53.585451 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:53.585459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:53.585682 (+ 223us) server_negotiation.cc:407] Connection header received
0926 02:53:53.585895 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:53.585900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:53.585972 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:53.586057 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:53.586916 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:53.587784 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:53.589405 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:53.590190 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:53.590345 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:53.599625 (+ 9280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:39125: BlockingRecv error: recv got EOF from 127.19.185.195:39125 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 26 02:53:53 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:53 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:53.654578 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:53.645677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:48635)
0926 02:53:53.646327 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:53.646344 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:53.646473 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:53.646860 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:53.646863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:53.647286 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:53.647527 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:53.647536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:53.648615 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:53.648619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:53.649678 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:53.649707 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:53.650624 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:53.650632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:53.650834 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:53.651516 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:53.651534 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:53.654320 (+ 2786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":321,"threads_started":1}
W20260926 02:53:53.654901 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:53.645819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:48635 (local address 127.19.185.254:39061)
0926 02:53:53.646044 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:53.646054 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:53.646437 (+ 383us) server_negotiation.cc:407] Connection header received
0926 02:53:53.646579 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:53.646583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:53.646684 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:53.646804 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:53.647672 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:53.648491 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:53.649809 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:53.650467 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:53.650604 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:53.654535 (+ 3931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:48635: BlockingRecv error: recv got EOF from 127.19.185.193:48635 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 02:53:54 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:54 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:54.126334 29518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:54.117001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:54085)
0926 02:53:54.117606 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:54.117622 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:54.117786 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:54.118241 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:54.118247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:54.118670 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:54.118838 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:54.118848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:54.119950 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:54.119962 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:54.120963 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:54.120976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:54.121736 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:54.121743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:54.121930 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:54.122681 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:54.122701 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:54.125977 (+ 3276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":279,"threads_started":1}
W20260926 02:53:54.126430 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:54.117118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:54085 (local address 127.19.185.254:39061)
0926 02:53:54.117393 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:54.117403 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:54.117714 (+ 311us) server_negotiation.cc:407] Connection header received
0926 02:53:54.117914 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:54.117919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:54.118001 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:54.118095 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:54.118988 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:54.119809 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:54.121103 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:54.121578 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:54.121703 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:54.126172 (+ 4469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:54085: BlockingRecv error: recv got EOF from 127.19.185.194:54085 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 02:53:54 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:54 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:54.609710 29519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:54.600658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:44695)
0926 02:53:54.601317 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:54.601334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:54.601520 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:54.601894 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:54.601899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:54.602385 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:54.602637 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:54.602651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:54.603852 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:54.603859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:54.605231 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:54.605245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:54.605972 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:54.605977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:54.606098 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:54.606679 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:54.606696 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:54.609468 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":308,"threads_started":1}
W20260926 02:53:54.609874 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:54.600761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:44695 (local address 127.19.185.254:39061)
0926 02:53:54.601136 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:54.601144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:54.601432 (+ 288us) server_negotiation.cc:407] Connection header received
0926 02:53:54.601668 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:54.601678 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:54.601742 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:54.601858 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:54.602802 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:54.603732 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:54.605374 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:54.605799 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:54.605900 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:54.609662 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:44695: BlockingRecv error: recv got EOF from 127.19.185.195:44695 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 02:53:54 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:54 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:54.663924 29520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:54.655577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:34587)
0926 02:53:54.656069 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:54.656085 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:54.656204 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:54.656499 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:54.656502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:54.656876 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:54.657019 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:54.657027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:54.658038 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:54.658042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:54.658976 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:54.658987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:54.659779 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:54.659783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:54.659889 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:54.660527 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:54.660547 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:54.663597 (+ 3050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":248,"threads_started":1}
W20260926 02:53:54.664311 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:54.655776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:34587 (local address 127.19.185.254:39061)
0926 02:53:54.655985 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:54.655993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:54.656158 (+ 165us) server_negotiation.cc:407] Connection header received
0926 02:53:54.656291 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:54.656295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:54.656369 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:54.656447 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:54.657178 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:54.657893 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:54.659136 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:54.659621 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:54.659732 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:54.663847 (+ 4115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:34587: BlockingRecv error: recv got EOF from 127.19.185.193:34587 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 02:53:55 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:55 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:55.135576 29521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:55.127382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:47701)
0926 02:53:55.127900 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:55.127916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:55.128025 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:55.128300 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:55.128303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:55.128673 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:55.128813 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:55.128821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:55.129938 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:55.129941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:55.130996 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:55.131008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:55.131795 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:55.131799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:55.131987 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:55.132548 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:55.132564 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:55.135273 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":278,"threads_started":1}
W20260926 02:53:55.135771 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:55.127463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:47701 (local address 127.19.185.254:39061)
0926 02:53:55.127624 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:55.127629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:55.127987 (+ 358us) server_negotiation.cc:407] Connection header received
0926 02:53:55.128118 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:55.128122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:55.128181 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:55.128260 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:55.128937 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:55.129814 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:55.131117 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:55.131605 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:55.131756 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:55.135508 (+ 3752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:47701: BlockingRecv error: recv got EOF from 127.19.185.194:47701 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 02:53:55 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:55 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:55.619810 29522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:55.610689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:53845)
0926 02:53:55.611389 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:55.611406 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:55.611528 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:55.611873 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:55.611877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:55.612333 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:55.612487 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:55.612495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:55.613742 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:55.613746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:55.614648 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:55.614661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:55.615473 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:55.615477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:55.615588 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:55.616231 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:55.616251 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:55.619484 (+ 3233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":411,"threads_started":1}
W20260926 02:53:55.619940 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:55.610742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:53845 (local address 127.19.185.254:39061)
0926 02:53:55.610924 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:55.610928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:55.611490 (+ 562us) server_negotiation.cc:407] Connection header received
0926 02:53:55.611672 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:55.611676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:55.611742 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:55.611828 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:55.612628 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:55.613585 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:55.614781 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:55.615261 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:55.615436 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:55.619690 (+ 4254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:53845: BlockingRecv error: recv got EOF from 127.19.185.195:53845 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 02:53:55 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:55 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:55.673717 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:55.664858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:53449)
0926 02:53:55.665392 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:55.665409 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:55.665564 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:55.665946 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:55.665950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:55.666336 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:55.666491 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:55.666499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:55.667557 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:55.667566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:55.668800 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:55.668813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:55.669844 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:55.669848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:55.669963 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:55.670608 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:55.670625 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:55.673455 (+ 2830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":302,"threads_started":1}
W20260926 02:53:55.673866 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:55.664938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:53449 (local address 127.19.185.254:39061)
0926 02:53:55.665137 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:55.665144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:55.665493 (+ 349us) server_negotiation.cc:407] Connection header received
0926 02:53:55.665702 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:55.665709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:55.665796 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:55.665900 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:55.666626 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:55.667442 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:55.668984 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:55.669703 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:55.669813 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:55.673659 (+ 3846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:53449: BlockingRecv error: recv got EOF from 127.19.185.193:53449 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 02:53:56 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:56 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:56.145802 29524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:56.136511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:53417)
0926 02:53:56.137041 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:56.137058 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:56.137252 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:56.137625 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:56.137630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:56.138095 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:56.138275 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:56.138284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:56.139430 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:56.139438 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:56.140503 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:56.140514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:56.141293 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:56.141300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:56.141510 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:56.142182 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:56.142202 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:56.145418 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":277,"threads_started":1}
W20260926 02:53:56.145964 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:56.136587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:53417 (local address 127.19.185.254:39061)
0926 02:53:56.136842 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:56.136850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:56.137149 (+ 299us) server_negotiation.cc:407] Connection header received
0926 02:53:56.137395 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:56.137401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:56.137476 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:56.137589 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:56.138430 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:56.139231 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:56.140665 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:56.141131 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:56.141291 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:56.145631 (+ 4340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:53417: BlockingRecv error: recv got EOF from 127.19.185.194:53417 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 26 02:53:56 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:56 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:56.629557 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:56.620808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:35129)
0926 02:53:56.621504 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:56.621520 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:56.621635 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:56.621965 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:56.621970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:56.622327 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:56.622559 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:56.622577 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:56.623745 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:56.623750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:56.624932 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:56.624945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:56.625728 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:56.625735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:56.625906 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:56.626533 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:56.626550 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:56.629316 (+ 2766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":428,"threads_started":1}
W20260926 02:53:56.629706 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:56.620855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:35129 (local address 127.19.185.254:39061)
0926 02:53:56.621134 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:56.621140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:56.621633 (+ 493us) server_negotiation.cc:407] Connection header received
0926 02:53:56.621742 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:56.621746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:56.621825 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:56.621903 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:56.622728 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:56.623633 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:56.625076 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:56.625560 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:56.625695 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:56.629493 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:35129: BlockingRecv error: recv got EOF from 127.19.185.195:35129 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Sep 26 02:53:56 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:56 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:56.683379 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:56.674805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:56425)
0926 02:53:56.675461 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:56.675477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:56.675621 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:56.675976 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:56.675979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:56.676346 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:56.676548 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:56.676558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:56.677614 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:56.677618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:56.678606 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:56.678623 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:56.679437 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:56.679442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:56.679566 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:56.680135 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:56.680153 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:56.683045 (+ 2892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":408,"threads_started":1}
W20260926 02:53:56.683741 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:56.674930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:56425 (local address 127.19.185.254:39061)
0926 02:53:56.675094 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:56.675098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:56.675597 (+ 499us) server_negotiation.cc:407] Connection header received
0926 02:53:56.675762 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:56.675766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:56.675847 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:56.675932 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:56.676708 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:56.677487 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:56.678764 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:56.679227 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:56.679410 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:56.683337 (+ 3927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:56425: BlockingRecv error: recv got EOF from 127.19.185.193:56425 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 02:53:57 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:57 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:57.155696 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:57.146824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:36195)
0926 02:53:57.147402 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:57.147417 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:57.147577 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:57.147869 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:57.147872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:57.148234 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:57.148381 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:57.148412 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:57.149419 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:57.149423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:57.150371 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:57.150382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:57.151194 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:57.151203 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:57.151415 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:57.152135 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:57.152157 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:57.155345 (+ 3188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":338,"threads_started":1}
W20260926 02:53:57.156033 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:57.146920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:36195 (local address 127.19.185.254:39061)
0926 02:53:57.147081 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:57.147085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:57.147559 (+ 474us) server_negotiation.cc:407] Connection header received
0926 02:53:57.147681 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:57.147685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:57.147748 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:57.147832 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:57.148561 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:57.149306 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:57.150547 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:57.150968 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:57.151078 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:57.155747 (+ 4669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:36195: BlockingRecv error: recv got EOF from 127.19.185.194:36195 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 02:53:57 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:57 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:57.638901 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:57.630549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:56005)
0926 02:53:57.631066 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:57.631082 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:57.631192 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:57.631556 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:57.631564 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:57.632003 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:57.632200 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:57.632209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:57.633604 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:57.633608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:57.634555 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:57.634566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:57.635202 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:57.635205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:57.635338 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:57.635905 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:57.635921 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:57.638663 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":262,"threads_started":1}
W20260926 02:53:57.639109 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:57.630761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:56005 (local address 127.19.185.254:39061)
0926 02:53:57.631015 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:57.631021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:57.631156 (+ 135us) server_negotiation.cc:407] Connection header received
0926 02:53:57.631306 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:57.631310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:57.631370 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:57.631451 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:57.632339 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:57.633459 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:57.634678 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:57.635077 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:57.635170 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:57.638844 (+ 3674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:56005: BlockingRecv error: recv got EOF from 127.19.185.195:56005 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Sep 26 02:53:57 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:57 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:57.693054 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:57.684310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:40373)
0926 02:53:57.684838 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:57.684857 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:57.685001 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:57.685466 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:57.685473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:57.685901 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:57.686069 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:57.686077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:57.687097 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:57.687104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:57.688051 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:57.688062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:57.688790 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:57.688796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:57.688966 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:57.689603 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:57.689621 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:57.692721 (+ 3100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":435,"threads_started":1}
W20260926 02:53:57.693231 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:57.684512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:40373 (local address 127.19.185.254:39061)
0926 02:53:57.684736 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:57.684743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:57.684919 (+ 176us) server_negotiation.cc:407] Connection header received
0926 02:53:57.685172 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:57.685218 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:57.685331 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:57.685423 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:57.686228 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:57.686960 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:57.688187 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:57.688641 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:57.688778 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:57.692961 (+ 4183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:40373: BlockingRecv error: recv got EOF from 127.19.185.193:40373 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 02:53:58 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:58 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:58.165024 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:58.156989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:34301)
0926 02:53:58.157529 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:58.157545 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:58.157651 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:58.157927 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:58.157929 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:58.158319 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:58.158502 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:58.158510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:58.159578 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:58.159582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:58.160569 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:58.160581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:58.161241 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:58.161245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:58.161360 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:58.161910 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:58.161926 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:58.164789 (+ 2863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":298,"threads_started":1}
W20260926 02:53:58.165218 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:58.157046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:34301 (local address 127.19.185.254:39061)
0926 02:53:58.157220 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:58.157224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:58.157615 (+ 391us) server_negotiation.cc:407] Connection header received
0926 02:53:58.157770 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:58.157773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:58.157832 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:58.157912 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:58.158641 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:58.159464 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:58.160693 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:58.161093 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:58.161213 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:58.164985 (+ 3772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:34301: BlockingRecv error: recv got EOF from 127.19.185.194:34301 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 02:53:58 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:58 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:58.651527 29531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:58.639864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:42379)
0926 02:53:58.640452 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:58.640469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:58.640579 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:58.640865 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:58.640868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:58.641290 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:58.641458 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:58.641466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:58.642469 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:58.642472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:58.643425 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:58.643437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:58.644174 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:58.644178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:58.644297 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:58.648307 (+ 4010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:58.648325 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:58.651213 (+ 2888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":360,"threads_started":1}
W20260926 02:53:58.651701 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:58.640012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:42379 (local address 127.19.185.254:39061)
0926 02:53:58.640190 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:58.640195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:58.640562 (+ 367us) server_negotiation.cc:407] Connection header received
0926 02:53:58.640667 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:58.640671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:58.640733 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:58.640817 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:58.641627 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:58.642352 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:58.643548 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:58.643950 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:58.644118 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:58.651463 (+ 7345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:42379: BlockingRecv error: recv got EOF from 127.19.185.195:42379 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 02:53:58 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:58 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:58.702690 29532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:58.694033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:60363)
0926 02:53:58.694561 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:58.694577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:58.694741 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:58.695174 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:58.695180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:58.695692 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:58.695880 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:58.695893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:58.697185 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:58.697193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:58.698217 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:58.698229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:58.698959 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:58.698967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:58.699154 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:58.699774 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:58.699791 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:58.702464 (+ 2673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":310,"threads_started":1}
W20260926 02:53:58.702932 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:58.694164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:60363 (local address 127.19.185.254:39061)
0926 02:53:58.694410 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:58.694416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:58.694669 (+ 253us) server_negotiation.cc:407] Connection header received
0926 02:53:58.694873 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:58.694878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:58.694945 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:58.695051 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:58.696018 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:58.697044 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:58.698365 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:58.698808 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:58.698944 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:58.702663 (+ 3719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:60363: BlockingRecv error: recv got EOF from 127.19.185.193:60363 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 26 02:53:59 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:59 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:59.174922 29533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:59.165917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:37333)
0926 02:53:59.166453 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:59.166473 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:59.166629 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:59.166977 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:59.166982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:59.167549 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:59.167773 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:59.167787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:59.168787 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:59.168793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:59.169904 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:59.169940 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:59.170738 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:59.170744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:59.170936 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:59.171643 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:59.171667 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:59.174595 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":261,"threads_started":1}
W20260926 02:53:59.175055 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:59.165967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:37333 (local address 127.19.185.254:39061)
0926 02:53:59.166364 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:59.166370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:59.166567 (+ 197us) server_negotiation.cc:407] Connection header received
0926 02:53:59.166736 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:59.166739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:59.166798 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:59.166906 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:59.167923 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:59.168633 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:59.170071 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:59.170584 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:59.170712 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:59.174822 (+ 4110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:37333: BlockingRecv error: recv got EOF from 127.19.185.194:37333 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 02:53:59 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:59 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:59.661393 29534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:59.652534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:59825)
0926 02:53:59.653078 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:59.653094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:59.653208 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:59.653616 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:59.653622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:59.654089 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:59.654260 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:59.654270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:59.655587 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:59.655594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:59.656617 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:59.656629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:59.657411 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:59.657418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:59.657628 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:59.658223 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:59.658241 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:59.661169 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":305,"threads_started":1}
W20260926 02:53:59.661758 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:59.652614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:59825 (local address 127.19.185.254:39061)
0926 02:53:59.652772 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:59.652776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:59.653168 (+ 392us) server_negotiation.cc:407] Connection header received
0926 02:53:59.653318 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:59.653322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:59.653381 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:59.653482 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:59.654431 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:59.655425 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:59.656772 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:59.657245 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:59.657395 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:59.661418 (+ 4023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:59825: BlockingRecv error: recv got EOF from 127.19.185.195:59825 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 02:53:59 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:53:59 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:53:59.711642 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:53:59.703610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:51027)
0926 02:53:59.704124 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 02:53:59.704140 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:53:59.704275 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:53:59.704553 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:53:59.704556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:53:59.704908 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:53:59.705063 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:59.705071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:59.706089 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:59.706092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:59.707101 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:53:59.707113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:53:59.707847 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:53:59.707851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:53:59.707965 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:59.708529 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:53:59.708546 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:53:59.711404 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":282,"threads_started":1}
W20260926 02:53:59.711845 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:53:59.703708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:51027 (local address 127.19.185.254:39061)
0926 02:53:59.703956 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 02:53:59.703963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:53:59.704235 (+ 272us) server_negotiation.cc:407] Connection header received
0926 02:53:59.704370 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:53:59.704374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:53:59.704434 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:53:59.704511 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:53:59.705217 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:59.705975 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:59.707241 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:53:59.707693 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:53:59.707787 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:53:59.711596 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:51027: BlockingRecv error: recv got EOF from 127.19.185.193:51027 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 02:54:00 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:54:00 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:54:00.184638 29536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:00.175870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:56377)
0926 02:54:00.176561 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:00.176582 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:00.176754 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:00.177106 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:00.177111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:00.177604 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:00.177758 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:00.177766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:00.178755 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:00.178762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:00.179696 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:00.179707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:00.180417 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:00.180423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:00.180589 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:00.181287 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:54:00.181304 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:00.184414 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":406,"threads_started":1}
W20260926 02:54:00.184860 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:54:00.176090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:56377 (local address 127.19.185.254:39061)
0926 02:54:00.176278 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:00.176287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:00.176705 (+ 418us) server_negotiation.cc:407] Connection header received
0926 02:54:00.176887 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:00.176893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:00.176971 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:00.177076 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:00.177899 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:00.178615 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:00.179837 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:00.180234 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:00.180362 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:00.184620 (+ 4258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:56377: BlockingRecv error: recv got EOF from 127.19.185.194:56377 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 02:54:00 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:54:00 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:54:00.670269 29537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:00.662314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:49491)
0926 02:54:00.662822 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:00.662839 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:00.662949 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:00.663232 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:00.663236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:00.663680 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:00.663835 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:00.663844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:00.664824 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:00.664828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:00.665812 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:00.665825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:00.666557 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:00.666561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:00.666677 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:00.667258 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:54:00.667274 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:00.670026 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":269,"threads_started":1}
W20260926 02:54:00.670467 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:54:00.662421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:49491 (local address 127.19.185.254:39061)
0926 02:54:00.662677 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:00.662683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:00.662916 (+ 233us) server_negotiation.cc:407] Connection header received
0926 02:54:00.663046 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:00.663049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:00.663108 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:00.663196 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:00.663965 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:00.664685 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:00.665949 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:00.666381 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:00.666475 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:00.670253 (+ 3778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:49491: BlockingRecv error: recv got EOF from 127.19.185.195:49491 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 26 02:54:00 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:54:00 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:54:00.721308 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:00.712624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:36931)
0926 02:54:00.713118 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:00.713134 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:00.713243 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:00.713577 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:00.713580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:00.713940 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:00.714206 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:00.714215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:00.715227 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:00.715232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:00.716246 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:00.716259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:00.717023 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:00.717031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:00.717168 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:00.717744 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:54:00.717768 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:00.720842 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":259,"threads_started":1}
W20260926 02:54:00.721338 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:54:00.712745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:36931 (local address 127.19.185.254:39061)
0926 02:54:00.712966 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:00.712971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:00.713208 (+ 237us) server_negotiation.cc:407] Connection header received
0926 02:54:00.713361 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:00.713364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:00.713454 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:00.713531 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:00.714329 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:00.715066 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:00.716381 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:00.716799 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:00.716895 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:00.721068 (+ 4173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:36931: BlockingRecv error: recv got EOF from 127.19.185.193:36931 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 26 02:54:01 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:54:01 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:54:01.194908 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:01.185673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:58981)
0926 02:54:01.186200 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:01.186215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:01.186369 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:01.186711 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:01.186718 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:01.187098 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:01.187278 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:01.187287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:01.188351 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:01.188356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:01.189280 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:01.189292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:01.190054 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:01.190060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:01.190228 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:01.190826 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:54:01.190844 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:01.194517 (+ 3673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":319,"threads_started":1}
W20260926 02:54:01.195096 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:54:01.185781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:58981 (local address 127.19.185.254:39061)
0926 02:54:01.186011 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:01.186018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:01.186298 (+ 280us) server_negotiation.cc:407] Connection header received
0926 02:54:01.186498 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:01.186502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:01.186561 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:01.186661 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:01.187473 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:01.188209 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:01.189418 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:01.189874 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:01.190007 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:01.194836 (+ 4829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:58981: BlockingRecv error: recv got EOF from 127.19.185.194:58981 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 02:54:01 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:54:01 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.195@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:54:01.679149 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:01.671251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.195:37185)
0926 02:54:01.671793 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:01.671816 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:01.671937 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:01.672303 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:01.672307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:01.672644 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:01.672814 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:01.672822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:01.673820 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:01.673824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:01.674745 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:01.674756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:01.675486 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:01.675490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:01.675614 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:01.676216 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:54:01.676232 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:01.678892 (+ 2660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":276,"threads_started":1}
W20260926 02:54:01.679487 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:54:01.671388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:37185 (local address 127.19.185.254:39061)
0926 02:54:01.671600 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:01.671604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:01.671902 (+ 298us) server_negotiation.cc:407] Connection header received
0926 02:54:01.672079 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:01.672083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:01.672158 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:01.672245 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:01.672943 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:01.673686 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:01.674898 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:01.675359 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:01.675460 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:01.679175 (+ 3715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.195:37185: BlockingRecv error: recv got EOF from 127.19.185.195:37185 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 02:54:01 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:54:01 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.193@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:54:01.731563 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:01.722451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.193:42915)
0926 02:54:01.723123 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:01.723140 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:01.723345 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:01.723764 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:01.723769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:01.724176 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:01.724329 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:01.724338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:01.725347 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:01.725353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:01.726392 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:01.726404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:01.727327 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:01.727336 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:01.727532 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:01.728291 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:54:01.728315 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:01.731240 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":307,"threads_started":1}
W20260926 02:54:01.731832 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:54:01.722751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:42915 (local address 127.19.185.254:39061)
0926 02:54:01.722953 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:01.722959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:01.723226 (+ 267us) server_negotiation.cc:407] Connection header received
0926 02:54:01.723519 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:01.723523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:01.723612 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:01.723722 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:01.724479 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:01.725204 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:01.726544 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:01.727126 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:01.727271 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:01.731555 (+ 4284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.193:42915: BlockingRecv error: recv got EOF from 127.19.185.193:42915 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 02:54:02 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:54:02 dist-test-slave-ktng krb5kdc[28497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.185.194@KRBTEST.COM for oryx/127.19.185.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:54:02.204288 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:02.196108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39061 (local address 127.19.185.194:55095)
0926 02:54:02.196663 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:02.196680 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:02.196794 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:02.197080 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:02.197083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:02.197439 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:02.197592 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:02.197600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:02.198588 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:02.198597 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:02.199784 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:02.199795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:02.200598 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:02.200604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:02.200766 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:02.201323 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:54:02.201339 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:02.204037 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.185.254:39061: Server oryx/127.19.185.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":302,"threads_started":1}
W20260926 02:54:02.204516 29233 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:54:02.196173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:55095 (local address 127.19.185.254:39061)
0926 02:54:02.196402 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:02.196412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:02.196780 (+ 368us) server_negotiation.cc:407] Connection header received
0926 02:54:02.196890 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:02.196893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:02.196953 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:02.197033 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:02.197741 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:02.198451 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:02.199917 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:02.200449 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:02.200577 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:02.204262 (+ 3685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.185.194:55095: BlockingRecv error: recv got EOF from 127.19.185.194:55095 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:54:02.514951 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 29084
I20260926 02:54:02.528209 29544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:02.516219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.19.185.193:46339)
0926 02:54:02.516742 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:02.516750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:02.516777 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:54:02.516838 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:02.516844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:02.517076 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:02.517234 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:02.518908 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:02.520138 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:02.521650 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:02.522364 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:02.522467 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:02.525100 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:02.525109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:02.525122 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:54:02.525158 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:02.527745 (+ 2587us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.185.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 02:54:02.527992 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42874: Request ticket server kudu/127.19.185.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":265,"threads_started":1}
W20260926 02:54:02.528415 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 29084: Not authorized: Client connection negotiation failed: client connection to 127.19.185.193:46339: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.185.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 02:54:02.528707 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 29084
I20260926 02:54:02.545204 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 29235
I20260926 02:54:02.557629 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:02.546244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.19.185.194:36579)
0926 02:54:02.546893 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:02.546900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:02.546930 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:54:02.546996 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:02.547003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:02.547246 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:02.547501 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:02.548796 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:02.549865 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:02.551145 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:02.551903 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:02.552019 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:02.554386 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:02.554394 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:02.554407 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:54:02.554438 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:02.557068 (+ 2630us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.185.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 02:54:02.557328 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40442: Request ticket server kudu/127.19.185.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":315,"threads_started":1}
W20260926 02:54:02.557746 29545 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40442: Request ticket server kudu/127.19.185.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 02:54:02.557734 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 29235: Not authorized: Client connection negotiation failed: client connection to 127.19.185.194:36579: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.185.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 02:54:02.557904 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 29235
I20260926 02:54:02.574290 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 29366
I20260926 02:54:02.585990 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:02.575322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50614 (local address 127.19.185.195:36509)
0926 02:54:02.575941 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:02.575949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:02.575980 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:54:02.576038 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:02.576045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:02.576299 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:02.576461 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:02.577684 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:02.578731 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:02.579990 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:02.580580 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:02.580722 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:02.583019 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:02.583026 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:02.583041 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:54:02.583079 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:02.585456 (+ 2377us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.185.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 02:54:02.585725 (+ 269us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50614: Request ticket server kudu/127.19.185.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":324,"threads_started":1}
W20260926 02:54:02.586077 29546 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50614: Request ticket server kudu/127.19.185.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 02:54:02.586539 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 29366: Not authorized: Client connection negotiation failed: client connection to 127.19.185.195:36509: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.185.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 02:54:02.586724 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 29366
I20260926 02:54:02.603205 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 29015
I20260926 02:54:02.604660 29077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:54:02.721359 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 29015
2026-09-26T02:54:02Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21170 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T02:54:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:54:02Z Disabled control of system clock
I20260926 02:54:02.798013 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:34159
--webserver_interface=127.19.185.254
--webserver_port=0
--builtin_ntp_servers=127.19.185.212:42527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:34159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 02:54:03.129333 29551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:54:03.129612 29551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:54:03.129778 29551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:54:03.137852 29551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:54:03.138046 29551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:54:03.138134 29551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:54:03.138248 29551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:54:03.138337 29551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:54:03.148543 29551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:42527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:34159
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:34159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:54:03.150023 29551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:54:03.151855 29551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:54:03.164865 29557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:03.165716 29556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:03.165825 29559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:54:03.166400 29551 server_base.cc:1034] running on GCE node
I20260926 02:54:03.167433 29551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:54:03.169338 29551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:54:03.170632 29551 hybrid_clock.cc:648] HybridClock initialized: now 1790391243170564 us; error 53 us; skew 500 ppm
I20260926 02:54:03.170972 29551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:54:03.173163 29551 webserver.cc:460] Webserver started at http://127.19.185.254:44987/ using document root <none> and password file <none>
I20260926 02:54:03.173830 29551 fs_manager.cc:359] Metadata directory not provided
I20260926 02:54:03.173916 29551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:54:03.174185 29551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:54:03.177018 29551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "39478f5720294a76b3a393cc81d4c29d"
format_stamp: "Formatted at 2026-09-26 02:54:03 on dist-test-slave-ktng"
server_key: "1d3d6c8dbe9bd15f590ed087644649cf"
server_key_iv: "7c49360525dbff19bc7c871afe5d1082"
server_key_version: "encryptionkey@0"
I20260926 02:54:03.177752 29551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "39478f5720294a76b3a393cc81d4c29d"
format_stamp: "Formatted at 2026-09-26 02:54:03 on dist-test-slave-ktng"
server_key: "1d3d6c8dbe9bd15f590ed087644649cf"
server_key_iv: "7c49360525dbff19bc7c871afe5d1082"
server_key_version: "encryptionkey@0"
I20260926 02:54:03.182812 29551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:54:03.186411 29565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:03.188040 29551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:54:03.188226 29551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "39478f5720294a76b3a393cc81d4c29d"
format_stamp: "Formatted at 2026-09-26 02:54:03 on dist-test-slave-ktng"
server_key: "1d3d6c8dbe9bd15f590ed087644649cf"
server_key_iv: "7c49360525dbff19bc7c871afe5d1082"
server_key_version: "encryptionkey@0"
I20260926 02:54:03.188443 29551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:54:03.217554 29551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:54:03.218506 29551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:54:03.218822 29551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:54:03.237757 29551 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:34159
I20260926 02:54:03.237776 29616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:34159 every 8 connection(s)
I20260926 02:54:03.239195 29551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:54:03.241451 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 29551
I20260926 02:54:03.241629 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:54:03.242282 20199 external_mini_cluster.cc:1428] Setting key 371746a794b1fb757324faad4e6c63e5
I20260926 02:54:03.243253 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:54:03.257161 29617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d: Bootstrap starting.
I20260926 02:54:03.257279 29620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:03.244308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.19.185.254:34159)
0926 02:54:03.245291 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:03.245304 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:03.245346 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:54:03.247766 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:03.247787 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:03.248267 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:03.248615 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:03.249322 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:03.250681 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:03.252530 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:03.253226 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:03.253371 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:03.254739 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:03.254750 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:03.254765 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:54:03.255469 (+ 704us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:03.255554 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:54:03.255577 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:03.255676 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:03.255900 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 02:54:03.256300 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":367,"threads_started":1}
I20260926 02:54:03.260455 29617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d: Neither blocks nor log segments found. Creating new log.
I20260926 02:54:03.261432 29617 log.cc:824] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:54:03.264220 29617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d: No bootstrap required, opened a new log
I20260926 02:54:03.271245 29617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39478f5720294a76b3a393cc81d4c29d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34159 } }
I20260926 02:54:03.271620 29617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:54:03.271731 29617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39478f5720294a76b3a393cc81d4c29d, State: Initialized, Role: FOLLOWER
I20260926 02:54:03.272331 29617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39478f5720294a76b3a393cc81d4c29d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34159 } }
I20260926 02:54:03.272532 29617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:54:03.272683 29617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:54:03.272859 29617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:54:03.274931 29617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39478f5720294a76b3a393cc81d4c29d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34159 } }
I20260926 02:54:03.275503 29617 leader_election.cc:302] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39478f5720294a76b3a393cc81d4c29d; no voters:
I20260926 02:54:03.276176 29617 leader_election.cc:288] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:54:03.276314 29622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:54:03.276965 29622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [term 1 LEADER]: Becoming Leader. State: Replica: 39478f5720294a76b3a393cc81d4c29d, State: Running, Role: LEADER
I20260926 02:54:03.277685 29622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39478f5720294a76b3a393cc81d4c29d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34159 } }
I20260926 02:54:03.278508 29617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:54:03.280581 29624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39478f5720294a76b3a393cc81d4c29d. Latest consensus state: current_term: 1 leader_uuid: "39478f5720294a76b3a393cc81d4c29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39478f5720294a76b3a393cc81d4c29d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34159 } } }
I20260926 02:54:03.280946 29624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [sys.catalog]: This master's current role is: LEADER
I20260926 02:54:03.281271 29623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39478f5720294a76b3a393cc81d4c29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39478f5720294a76b3a393cc81d4c29d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34159 } } }
I20260926 02:54:03.281729 29623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d [sys.catalog]: This master's current role is: LEADER
I20260926 02:54:03.283146 29628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:54:03.289604 29628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:54:03.302572 29628 catalog_manager.cc:1347] Generated new cluster ID: 6bc17e7875e3438497445e8f9394406c
I20260926 02:54:03.302714 29628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:54:03.335352 29628 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:54:03.336902 29628 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:54:03.344991 29628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39478f5720294a76b3a393cc81d4c29d: Generated new TSK 0
I20260926 02:54:03.345856 29628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:54:03.355975 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34159
--builtin_ntp_servers=127.19.185.212:42527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 02:54:03.688792 29641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:54:03.689108 29641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:54:03.689267 29641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:54:03.696834 29641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:54:03.696938 29641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:54:03.697098 29641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:54:03.707420 29641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:42527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:54:03.708839 29641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:54:03.710872 29641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:54:03.724983 29646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:03.725047 29649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:03.724963 29647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:54:03.725759 29641 server_base.cc:1034] running on GCE node
I20260926 02:54:03.726403 29641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:54:03.727421 29641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:54:03.728681 29641 hybrid_clock.cc:648] HybridClock initialized: now 1790391243728617 us; error 62 us; skew 500 ppm
I20260926 02:54:03.729035 29641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:54:03.737427 29641 webserver.cc:460] Webserver started at http://127.19.185.193:43245/ using document root <none> and password file <none>
I20260926 02:54:03.738242 29641 fs_manager.cc:359] Metadata directory not provided
I20260926 02:54:03.738382 29641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:54:03.738730 29641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:54:03.741701 29641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "5eb0be36cf9e4476ab48b1dd3c6cb0bd"
format_stamp: "Formatted at 2026-09-26 02:54:03 on dist-test-slave-ktng"
server_key: "ee14c6b1095651584eaa6aa30e7cbae1"
server_key_iv: "3700cd6a032f1d907a95cbbf8e8677c4"
server_key_version: "encryptionkey@0"
I20260926 02:54:03.742499 29641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "5eb0be36cf9e4476ab48b1dd3c6cb0bd"
format_stamp: "Formatted at 2026-09-26 02:54:03 on dist-test-slave-ktng"
server_key: "ee14c6b1095651584eaa6aa30e7cbae1"
server_key_iv: "3700cd6a032f1d907a95cbbf8e8677c4"
server_key_version: "encryptionkey@0"
I20260926 02:54:03.747836 29641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:54:03.751716 29655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:03.753711 29641 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:54:03.753937 29641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "5eb0be36cf9e4476ab48b1dd3c6cb0bd"
format_stamp: "Formatted at 2026-09-26 02:54:03 on dist-test-slave-ktng"
server_key: "ee14c6b1095651584eaa6aa30e7cbae1"
server_key_iv: "3700cd6a032f1d907a95cbbf8e8677c4"
server_key_version: "encryptionkey@0"
I20260926 02:54:03.754225 29641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:54:03.770876 29641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:54:03.771945 29641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:54:03.772360 29641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:54:03.773324 29641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:54:03.774770 29641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:54:03.774892 29641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:03.775027 29641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:54:03.775117 29641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:03.807574 29641 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:43927
I20260926 02:54:03.807608 29767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:43927 every 8 connection(s)
I20260926 02:54:03.809077 29641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:54:03.811385 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 29641
I20260926 02:54:03.811570 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:54:03.812417 20199 external_mini_cluster.cc:1428] Setting key c43eec9b237c7b7264804089245690cb
I20260926 02:54:03.815937 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34159
--builtin_ntp_servers=127.19.185.212:42527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 02:54:03.827214 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:03.813684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:37885 (local address 127.19.185.254:34159)
0926 02:54:03.814158 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:03.814166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:03.815746 (+ 1580us) server_negotiation.cc:407] Connection header received
0926 02:54:03.818785 (+ 3039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:03.818789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:03.818855 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:03.818948 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:03.820507 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:03.821354 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:03.823910 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:03.824414 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:03.824525 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:03.826146 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:03.826155 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:03.826164 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:54:03.826192 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:03.826249 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:54:03.826270 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:03.826368 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:03.826619 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 02:54:03.826740 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":232,"threads_started":1}
I20260926 02:54:03.828063 29771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:03.814281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:34159 (local address 127.19.185.193:37885)
0926 02:54:03.815552 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:03.815618 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:03.818447 (+ 2829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:03.819237 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:03.819251 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:03.819415 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:03.820283 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:03.820310 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:03.821530 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:03.821538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:03.823754 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:03.823772 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:03.824637 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:03.824642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:03.824873 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:03.825745 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:54:03.825747 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:03.825790 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:54:03.825872 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:54:03.826380 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:54:03.826389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:54:03.826395 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:54:03.826549 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 02:54:03.826859 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":408,"threads_started":1}
I20260926 02:54:03.829814 29768 heartbeater.cc:347] Connected to a master server at 127.19.185.254:34159
I20260926 02:54:03.830188 29768 heartbeater.cc:464] Registering TS with master...
I20260926 02:54:03.831209 29768 heartbeater.cc:511] Master 127.19.185.254:34159 requested a full tablet report, sending...
I20260926 02:54:03.833653 29582 ts_manager.cc:194] Registered new tserver with Master: 5eb0be36cf9e4476ab48b1dd3c6cb0bd (127.19.185.193:43927)
I20260926 02:54:03.835280 29582 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.185.193:37885
W20260926 02:54:04.152218 29772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:54:04.152537 29772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:54:04.152680 29772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:54:04.159973 29772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:54:04.160099 29772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:54:04.160317 29772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:54:04.169384 29772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:42527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:54:04.170639 29772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:54:04.172443 29772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:54:04.185633 29777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:04.185833 29778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:04.185777 29780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:54:04.186884 29772 server_base.cc:1034] running on GCE node
I20260926 02:54:04.187566 29772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:54:04.188491 29772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:54:04.189747 29772 hybrid_clock.cc:648] HybridClock initialized: now 1790391244189714 us; error 41 us; skew 500 ppm
I20260926 02:54:04.190243 29772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:54:04.193071 29772 webserver.cc:460] Webserver started at http://127.19.185.194:41351/ using document root <none> and password file <none>
I20260926 02:54:04.193796 29772 fs_manager.cc:359] Metadata directory not provided
I20260926 02:54:04.193926 29772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:54:04.194294 29772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:54:04.197366 29772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "25cf58b1dcbd429ca1f0d5c5573a175e"
format_stamp: "Formatted at 2026-09-26 02:54:04 on dist-test-slave-ktng"
server_key: "e552458f10fbc2e9c4e8fc1b096acdad"
server_key_iv: "ef3ef8b7ce262f6f8db216a31feb6a7f"
server_key_version: "encryptionkey@0"
I20260926 02:54:04.198235 29772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "25cf58b1dcbd429ca1f0d5c5573a175e"
format_stamp: "Formatted at 2026-09-26 02:54:04 on dist-test-slave-ktng"
server_key: "e552458f10fbc2e9c4e8fc1b096acdad"
server_key_iv: "ef3ef8b7ce262f6f8db216a31feb6a7f"
server_key_version: "encryptionkey@0"
I20260926 02:54:04.203856 29772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 02:54:04.207581 29786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:04.208951 29772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:54:04.209187 29772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "25cf58b1dcbd429ca1f0d5c5573a175e"
format_stamp: "Formatted at 2026-09-26 02:54:04 on dist-test-slave-ktng"
server_key: "e552458f10fbc2e9c4e8fc1b096acdad"
server_key_iv: "ef3ef8b7ce262f6f8db216a31feb6a7f"
server_key_version: "encryptionkey@0"
I20260926 02:54:04.209414 29772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:54:04.238638 29772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:54:04.239593 29772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:54:04.239895 29772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:54:04.240876 29772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:54:04.242347 29772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:54:04.242470 29772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:04.242609 29772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:54:04.242699 29772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:04.279275 29772 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:38031
I20260926 02:54:04.279400 29898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:38031 every 8 connection(s)
I20260926 02:54:04.280683 29772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:54:04.284819 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 29772
I20260926 02:54:04.284993 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:54:04.285596 20199 external_mini_cluster.cc:1428] Setting key cf786fa53ad1e8c3eec2d6312340e787
I20260926 02:54:04.288480 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34159
--builtin_ntp_servers=127.19.185.212:42527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 02:54:04.294160 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:04.283284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:44039 (local address 127.19.185.254:34159)
0926 02:54:04.283526 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:04.283532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:04.285066 (+ 1534us) server_negotiation.cc:407] Connection header received
0926 02:54:04.287278 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:04.287282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:04.287410 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:04.287512 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:04.288678 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:04.289425 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:04.291113 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:04.291648 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:04.291781 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:04.293259 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:04.293268 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:04.293275 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:54:04.293301 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:04.293345 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:54:04.293362 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:04.293455 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:04.293653 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 02:54:04.293769 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:54:04.295032 29901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:04.283797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:34159 (local address 127.19.185.194:44039)
0926 02:54:04.284874 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:04.284932 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:04.286986 (+ 2054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:04.287789 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:04.287803 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:04.287892 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:04.288507 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:04.288523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:04.289574 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:04.289581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:04.290974 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:04.290986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:04.291889 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:04.291899 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:04.292174 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:04.292913 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:54:04.292916 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:04.292956 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:54:04.293034 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:54:04.293459 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:54:04.293467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:54:04.293472 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 02:54:04.293615 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 02:54:04.293932 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":275,"threads_started":1}
I20260926 02:54:04.296654 29899 heartbeater.cc:347] Connected to a master server at 127.19.185.254:34159
I20260926 02:54:04.296969 29899 heartbeater.cc:464] Registering TS with master...
I20260926 02:54:04.297878 29899 heartbeater.cc:511] Master 127.19.185.254:34159 requested a full tablet report, sending...
I20260926 02:54:04.299621 29582 ts_manager.cc:194] Registered new tserver with Master: 25cf58b1dcbd429ca1f0d5c5573a175e (127.19.185.194:38031)
I20260926 02:54:04.300447 29582 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.185.194:44039
W20260926 02:54:04.615988 29902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:54:04.616312 29902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:54:04.616473 29902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:54:04.623940 29902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:54:04.624071 29902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:54:04.624246 29902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:54:04.633318 29902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:42527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:54:04.634603 29902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:54:04.636385 29902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:54:04.649374 29907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:04.649394 29908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:04.649336 29910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:54:04.649756 29902 server_base.cc:1034] running on GCE node
I20260926 02:54:04.650677 29902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:54:04.651741 29902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:54:04.653007 29902 hybrid_clock.cc:648] HybridClock initialized: now 1790391244652942 us; error 56 us; skew 500 ppm
I20260926 02:54:04.653379 29902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:54:04.655755 29902 webserver.cc:460] Webserver started at http://127.19.185.195:46051/ using document root <none> and password file <none>
I20260926 02:54:04.656514 29902 fs_manager.cc:359] Metadata directory not provided
I20260926 02:54:04.656658 29902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:54:04.657013 29902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:54:04.660243 29902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "9f6933cf37294c8eb5817dac02abd849"
format_stamp: "Formatted at 2026-09-26 02:54:04 on dist-test-slave-ktng"
server_key: "e23e43e63a674cc58c7a8211e0b7e609"
server_key_iv: "721997b093e66c31507e2af3f05d4f9c"
server_key_version: "encryptionkey@0"
I20260926 02:54:04.661139 29902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "9f6933cf37294c8eb5817dac02abd849"
format_stamp: "Formatted at 2026-09-26 02:54:04 on dist-test-slave-ktng"
server_key: "e23e43e63a674cc58c7a8211e0b7e609"
server_key_iv: "721997b093e66c31507e2af3f05d4f9c"
server_key_version: "encryptionkey@0"
I20260926 02:54:04.666676 29902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:54:04.670596 29916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:04.672096 29902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:54:04.672289 29902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "9f6933cf37294c8eb5817dac02abd849"
format_stamp: "Formatted at 2026-09-26 02:54:04 on dist-test-slave-ktng"
server_key: "e23e43e63a674cc58c7a8211e0b7e609"
server_key_iv: "721997b093e66c31507e2af3f05d4f9c"
server_key_version: "encryptionkey@0"
I20260926 02:54:04.672521 29902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:54:04.685730 29902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:54:04.686590 29902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:54:04.686906 29902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:54:04.687849 29902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:54:04.689260 29902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:54:04.689380 29902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:04.689510 29902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:54:04.689597 29902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:04.722467 29902 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:38369
I20260926 02:54:04.722558 30028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:38369 every 8 connection(s)
I20260926 02:54:04.723984 29902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:54:04.733974 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 29902
I20260926 02:54:04.734143 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:54:04.734757 20199 external_mini_cluster.cc:1428] Setting key c81469cc104d66efa650a83bca9dcc23
I20260926 02:54:04.736996 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:04.726679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:60849 (local address 127.19.185.254:34159)
0926 02:54:04.726902 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:04.726908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:04.728327 (+ 1419us) server_negotiation.cc:407] Connection header received
0926 02:54:04.729998 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:04.730002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:04.730072 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:04.730164 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:04.731247 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:04.732084 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:04.733635 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:04.734280 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:04.734413 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:04.736122 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:04.736128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:04.736132 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:54:04.736153 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:04.736201 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:54:04.736219 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:04.736337 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:04.736566 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 02:54:04.736671 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:54:04.738391 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:04.727216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:34159 (local address 127.19.185.195:60849)
0926 02:54:04.728173 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:04.728209 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:04.729739 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:04.730432 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:04.730439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:04.730525 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:04.731078 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:04.731093 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:04.732260 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:04.732264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:04.733456 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:04.733467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:04.734489 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:04.734494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:04.734973 (+ 479us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:04.735736 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:54:04.735739 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:04.735780 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:54:04.735916 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:54:04.736345 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:54:04.736354 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:54:04.736361 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 02:54:04.736523 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 02:54:04.736927 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":282,"threads_started":1}
I20260926 02:54:04.739821 30029 heartbeater.cc:347] Connected to a master server at 127.19.185.254:34159
I20260926 02:54:04.740132 30029 heartbeater.cc:464] Registering TS with master...
I20260926 02:54:04.740859 30029 heartbeater.cc:511] Master 127.19.185.254:34159 requested a full tablet report, sending...
I20260926 02:54:04.742183 29582 ts_manager.cc:194] Registered new tserver with Master: 9f6933cf37294c8eb5817dac02abd849 (127.19.185.195:38369)
I20260926 02:54:04.742902 29582 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.185.195:60849
I20260926 02:54:04.750331 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 02:54:04.756598 29770 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:54:04.754889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.19.185.254:34159)
0926 02:54:04.755083 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:04.755087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:04.755613 (+ 526us) server_negotiation.cc:407] Connection header received
0926 02:54:04.755823 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:04.755834 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:04.755907 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:04.756014 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:04.756417 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42046: BlockingRecv error: recv got EOF from 127.0.0.1:42046 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:54:04.757436 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 29641
I20260926 02:54:04.765373 30038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:04.758613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42346 (local address 127.19.185.193:43927)
0926 02:54:04.759130 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:04.759137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:04.759163 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:54:04.759222 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:04.759228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:04.759504 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:04.759634 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:04.760027 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:04.761077 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:04.762317 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:04.762855 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:04.762948 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:04.763977 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:04.763984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:04.763997 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:54:04.764581 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:04.764629 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:54:04.764646 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:04.764744 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:04.764886 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 02:54:04.765031 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":285,"threads_started":1}
I20260926 02:54:04.766568 29763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:54:04.848589 29768 heartbeater.cc:503] Master 127.19.185.254:34159 was elected leader, sending a full tablet report...
I20260926 02:54:04.849984 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 29641
W20260926 02:54:04.866848 29568 connection.cc:535] server connection from 127.19.185.193:37885 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 02:54:04.867268 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 29772
I20260926 02:54:04.875196 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:04.868210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43104 (local address 127.19.185.194:38031)
0926 02:54:04.868786 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:04.868793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:04.868819 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:54:04.868883 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:04.868890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:04.869099 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:04.869275 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:04.869673 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:04.870652 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:04.871954 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:04.872499 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:04.872644 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:04.873676 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:04.873684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:04.873697 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:54:04.874361 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:04.874408 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:54:04.874438 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:04.874573 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:04.874724 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 02:54:04.874861 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":322,"threads_started":1}
I20260926 02:54:04.876513 29894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:54:04.959156 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 29772
I20260926 02:54:04.975457 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 29902
I20260926 02:54:04.984072 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:04.976582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.19.185.195:38369)
0926 02:54:04.977134 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:04.977148 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:04.977179 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:54:04.977248 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:04.977255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:04.977469 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:04.977598 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:04.977974 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:04.979121 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:04.980723 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:04.981293 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:04.981398 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:04.982395 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:04.982403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:04.982417 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:54:04.983025 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:04.983074 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:54:04.983094 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:04.983200 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:04.983537 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 02:54:04.983651 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":265,"threads_started":1}
I20260926 02:54:04.985432 30024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:54:05.070535 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 29902
I20260926 02:54:05.086826 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 29551
I20260926 02:54:05.088085 29612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:54:05.194180 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 29551
2026-09-26T02:54:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2467 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T02:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:54:05Z Disabled control of system clock
I20260926 02:54:05.265187 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:37909
--webserver_interface=127.19.185.254
--webserver_port=0
--builtin_ntp_servers=127.19.185.212:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:37909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 02:54:05.586567 30048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:54:05.586858 30048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:54:05.586979 30048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:54:05.594223 30048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:54:05.594338 30048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:54:05.594408 30048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:54:05.594523 30048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:54:05.594602 30048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:54:05.603925 30048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:37909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:37909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:54:05.605361 30048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:54:05.607156 30048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:54:05.618997 30053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:05.619195 30054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:05.619320 30056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:54:05.620151 30048 server_base.cc:1034] running on GCE node
I20260926 02:54:05.620926 30048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:54:05.622313 30048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:54:05.623535 30048 hybrid_clock.cc:648] HybridClock initialized: now 1790391245623496 us; error 78 us; skew 500 ppm
I20260926 02:54:05.623878 30048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:54:05.626176 30048 webserver.cc:460] Webserver started at http://127.19.185.254:46449/ using document root <none> and password file <none>
I20260926 02:54:05.626868 30048 fs_manager.cc:359] Metadata directory not provided
I20260926 02:54:05.627000 30048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:54:05.627370 30048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:54:05.631918 30048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "8465988c5399480dafa851f3efb154a8"
format_stamp: "Formatted at 2026-09-26 02:54:05 on dist-test-slave-ktng"
server_key: "2f5c60ef691dd44658ac79105de91da0"
server_key_iv: "bff6900aeb87db0a0e379f0b15e3c447"
server_key_version: "encryptionkey@0"
I20260926 02:54:05.632695 30048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "8465988c5399480dafa851f3efb154a8"
format_stamp: "Formatted at 2026-09-26 02:54:05 on dist-test-slave-ktng"
server_key: "2f5c60ef691dd44658ac79105de91da0"
server_key_iv: "bff6900aeb87db0a0e379f0b15e3c447"
server_key_version: "encryptionkey@0"
I20260926 02:54:05.638103 30048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:54:05.641441 30062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:05.642864 30048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:54:05.643042 30048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "8465988c5399480dafa851f3efb154a8"
format_stamp: "Formatted at 2026-09-26 02:54:05 on dist-test-slave-ktng"
server_key: "2f5c60ef691dd44658ac79105de91da0"
server_key_iv: "bff6900aeb87db0a0e379f0b15e3c447"
server_key_version: "encryptionkey@0"
I20260926 02:54:05.643263 30048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:54:05.648877 30048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:54:05.649603 30048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:54:05.649904 30048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:54:05.668185 30048 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:37909
I20260926 02:54:05.668217 30113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:37909 every 8 connection(s)
I20260926 02:54:05.669504 30048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:54:05.673213 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:54:05.676494 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 30048
I20260926 02:54:05.676702 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:54:05.677332 20199 external_mini_cluster.cc:1428] Setting key 05764ac54337fe6c7286533a77c3378a
I20260926 02:54:05.686089 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:05.679242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.19.185.254:37909)
0926 02:54:05.680323 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:05.680337 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:05.680389 (+ 52us) server_negotiation.cc:407] Connection header received
0926 02:54:05.682109 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:05.682142 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:05.682689 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:05.683125 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:05.684161 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:05.684173 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:05.684194 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:54:05.684216 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:05.684287 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:54:05.684313 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:05.684538 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:05.684681 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 02:54:05.684925 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":315,"threads_started":1}
I20260926 02:54:05.686720 30114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8: Bootstrap starting.
I20260926 02:54:05.689795 30114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8: Neither blocks nor log segments found. Creating new log.
I20260926 02:54:05.690789 30114 log.cc:824] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8: Log is configured to *not* fsync() on all Append() calls
I20260926 02:54:05.693749 30114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8: No bootstrap required, opened a new log
I20260926 02:54:05.701454 30114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8465988c5399480dafa851f3efb154a8" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37909 } }
I20260926 02:54:05.701848 30114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:54:05.701956 30114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8465988c5399480dafa851f3efb154a8, State: Initialized, Role: FOLLOWER
I20260926 02:54:05.702509 30114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8465988c5399480dafa851f3efb154a8" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37909 } }
I20260926 02:54:05.702746 30114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:54:05.702889 30114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:54:05.703061 30114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:54:05.705529 30114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8465988c5399480dafa851f3efb154a8" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37909 } }
I20260926 02:54:05.706054 30114 leader_election.cc:302] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8465988c5399480dafa851f3efb154a8; no voters:
I20260926 02:54:05.706562 30114 leader_election.cc:288] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:54:05.706703 30119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:54:05.707715 30119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [term 1 LEADER]: Becoming Leader. State: Replica: 8465988c5399480dafa851f3efb154a8, State: Running, Role: LEADER
I20260926 02:54:05.708256 30119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8465988c5399480dafa851f3efb154a8" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37909 } }
I20260926 02:54:05.708709 30114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:54:05.710942 30121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8465988c5399480dafa851f3efb154a8. Latest consensus state: current_term: 1 leader_uuid: "8465988c5399480dafa851f3efb154a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8465988c5399480dafa851f3efb154a8" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37909 } } }
I20260926 02:54:05.711325 30121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [sys.catalog]: This master's current role is: LEADER
I20260926 02:54:05.711815 30120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8465988c5399480dafa851f3efb154a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8465988c5399480dafa851f3efb154a8" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37909 } } }
I20260926 02:54:05.712112 30120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8 [sys.catalog]: This master's current role is: LEADER
I20260926 02:54:05.716398 30126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:54:05.721577 30126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:54:05.730221 30126 catalog_manager.cc:1347] Generated new cluster ID: 4e805f23659245c4b9a915c4c8ec5487
I20260926 02:54:05.730324 30126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:54:05.747123 30126 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:54:05.747715 30126 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:54:05.757812 30126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8465988c5399480dafa851f3efb154a8: Generated new TSK 0
I20260926 02:54:05.758698 30126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:54:05.767452 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37909
--builtin_ntp_servers=127.19.185.212:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 02:54:06.116320 30138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:54:06.116624 30138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:54:06.116794 30138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:54:06.124017 30138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:54:06.124182 30138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:54:06.124341 30138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:54:06.133342 30138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:54:06.134639 30138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:54:06.136473 30138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:54:06.150295 30143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:06.150451 30144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:06.150269 30146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:54:06.151096 30138 server_base.cc:1034] running on GCE node
I20260926 02:54:06.151829 30138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:54:06.152871 30138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:54:06.154191 30138 hybrid_clock.cc:648] HybridClock initialized: now 1790391246154148 us; error 59 us; skew 500 ppm
I20260926 02:54:06.154579 30138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:54:06.157064 30138 webserver.cc:460] Webserver started at http://127.19.185.193:46151/ using document root <none> and password file <none>
I20260926 02:54:06.157837 30138 fs_manager.cc:359] Metadata directory not provided
I20260926 02:54:06.157990 30138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:54:06.158315 30138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:54:06.161337 30138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "4e2e6b5689dd4136becc71f6e3dcb517"
format_stamp: "Formatted at 2026-09-26 02:54:06 on dist-test-slave-ktng"
server_key: "79dc71a40e2181dd330f53d4f8db7848"
server_key_iv: "0c6cdf5a68d37027186dfe2d8cf5dd9c"
server_key_version: "encryptionkey@0"
I20260926 02:54:06.162125 30138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "4e2e6b5689dd4136becc71f6e3dcb517"
format_stamp: "Formatted at 2026-09-26 02:54:06 on dist-test-slave-ktng"
server_key: "79dc71a40e2181dd330f53d4f8db7848"
server_key_iv: "0c6cdf5a68d37027186dfe2d8cf5dd9c"
server_key_version: "encryptionkey@0"
I20260926 02:54:06.167150 30138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:54:06.170581 30152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:06.171885 30138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:54:06.172056 30138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "4e2e6b5689dd4136becc71f6e3dcb517"
format_stamp: "Formatted at 2026-09-26 02:54:06 on dist-test-slave-ktng"
server_key: "79dc71a40e2181dd330f53d4f8db7848"
server_key_iv: "0c6cdf5a68d37027186dfe2d8cf5dd9c"
server_key_version: "encryptionkey@0"
I20260926 02:54:06.172284 30138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:54:06.177721 30138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:54:06.178489 30138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:54:06.178759 30138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:54:06.179654 30138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:54:06.180943 30138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:54:06.181069 30138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:06.181193 30138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:54:06.181278 30138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:06.214412 30138 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:37045
I20260926 02:54:06.214455 30264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:37045 every 8 connection(s)
I20260926 02:54:06.215914 30138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:54:06.224471 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:06.218741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:39423 (local address 127.19.185.254:37909)
0926 02:54:06.219229 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:06.219234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:06.220446 (+ 1212us) server_negotiation.cc:407] Connection header received
0926 02:54:06.222190 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:06.222196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:06.222270 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:06.222398 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:06.223846 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:06.223850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:06.223853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:54:06.223869 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:06.223910 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:54:06.223922 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:06.224030 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:06.224213 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 02:54:06.224262 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":261,"threads_started":1}
I20260926 02:54:06.225435 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:06.219359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:37909 (local address 127.19.185.193:39423)
0926 02:54:06.220282 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:06.220326 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:06.221916 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:06.222645 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:06.222653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:06.222745 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:06.223538 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:54:06.223540 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:06.223579 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:54:06.223664 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:54:06.224017 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:54:06.224032 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:54:06.224038 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:54:06.224257 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 02:54:06.224455 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":291,"threads_started":1}
I20260926 02:54:06.226567 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 30138
I20260926 02:54:06.226800 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:54:06.227224 30265 heartbeater.cc:347] Connected to a master server at 127.19.185.254:37909
I20260926 02:54:06.227530 20199 external_mini_cluster.cc:1428] Setting key 53f65b8e240babf7192579fed2f15262
I20260926 02:54:06.227644 30265 heartbeater.cc:464] Registering TS with master...
I20260926 02:54:06.228384 30265 heartbeater.cc:511] Master 127.19.185.254:37909 requested a full tablet report, sending...
I20260926 02:54:06.230844 30079 ts_manager.cc:194] Registered new tserver with Master: 4e2e6b5689dd4136becc71f6e3dcb517 (127.19.185.193:37045)
I20260926 02:54:06.230890 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37909
--builtin_ntp_servers=127.19.185.212:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 02:54:06.556336 30269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:54:06.556661 30269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:54:06.556823 30269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:54:06.564023 30269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:54:06.564153 30269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:54:06.564343 30269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:54:06.573947 30269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:54:06.575309 30269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:54:06.577278 30269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:54:06.590619 30277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:06.590673 30275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:06.591217 30274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:54:06.591539 30269 server_base.cc:1034] running on GCE node
I20260926 02:54:06.592250 30269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:54:06.593271 30269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:54:06.594542 30269 hybrid_clock.cc:648] HybridClock initialized: now 1790391246594531 us; error 67 us; skew 500 ppm
I20260926 02:54:06.594933 30269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:54:06.597678 30269 webserver.cc:460] Webserver started at http://127.19.185.194:38067/ using document root <none> and password file <none>
I20260926 02:54:06.598562 30269 fs_manager.cc:359] Metadata directory not provided
I20260926 02:54:06.598711 30269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:54:06.599071 30269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:54:06.602209 30269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "59f29b42a39643089f2c96f34df91dd1"
format_stamp: "Formatted at 2026-09-26 02:54:06 on dist-test-slave-ktng"
server_key: "65e9e7480124298987feedf4b2ef75f3"
server_key_iv: "51687c6841f1b1d932083718538d6345"
server_key_version: "encryptionkey@0"
I20260926 02:54:06.603066 30269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "59f29b42a39643089f2c96f34df91dd1"
format_stamp: "Formatted at 2026-09-26 02:54:06 on dist-test-slave-ktng"
server_key: "65e9e7480124298987feedf4b2ef75f3"
server_key_iv: "51687c6841f1b1d932083718538d6345"
server_key_version: "encryptionkey@0"
I20260926 02:54:06.608669 30269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:54:06.612612 30283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:06.614167 30269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:54:06.614367 30269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "59f29b42a39643089f2c96f34df91dd1"
format_stamp: "Formatted at 2026-09-26 02:54:06 on dist-test-slave-ktng"
server_key: "65e9e7480124298987feedf4b2ef75f3"
server_key_iv: "51687c6841f1b1d932083718538d6345"
server_key_version: "encryptionkey@0"
I20260926 02:54:06.614598 30269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:54:06.620498 30269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:54:06.621305 30269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:54:06.621582 30269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:54:06.622586 30269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:54:06.624079 30269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:54:06.624200 30269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:06.624336 30269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:54:06.624428 30269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:06.657861 30269 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:40567
I20260926 02:54:06.657905 30395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:40567 every 8 connection(s)
I20260926 02:54:06.659255 30269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:54:06.663940 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 30269
I20260926 02:54:06.664129 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:54:06.664719 20199 external_mini_cluster.cc:1428] Setting key 4fc3cd622b0e03a3add4c7de98c55fd9
I20260926 02:54:06.667558 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37909
--builtin_ntp_servers=127.19.185.212:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 02:54:06.668017 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:06.661838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:42495 (local address 127.19.185.254:37909)
0926 02:54:06.662090 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:06.662095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:06.663387 (+ 1292us) server_negotiation.cc:407] Connection header received
0926 02:54:06.665716 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:06.665722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:06.665818 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:06.665940 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:06.667259 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:06.667265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:06.667270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:54:06.667287 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:06.667346 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:54:06.667362 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:06.667477 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:06.667659 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 02:54:06.667727 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:54:06.668993 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:06.662256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:37909 (local address 127.19.185.194:42495)
0926 02:54:06.663204 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:06.663240 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:06.665430 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:06.666184 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:06.666192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:06.666285 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:06.666963 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:54:06.666972 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:06.667009 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:54:06.667093 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:54:06.667441 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:54:06.667451 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:54:06.667458 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 02:54:06.667830 (+ 372us) client_negotiation.cc:240] Negotiation successful
0926 02:54:06.668109 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":271,"threads_started":1}
I20260926 02:54:06.676347 30396 heartbeater.cc:347] Connected to a master server at 127.19.185.254:37909
I20260926 02:54:06.676687 30396 heartbeater.cc:464] Registering TS with master...
I20260926 02:54:06.677430 30396 heartbeater.cc:511] Master 127.19.185.254:37909 requested a full tablet report, sending...
I20260926 02:54:06.679014 30079 ts_manager.cc:194] Registered new tserver with Master: 59f29b42a39643089f2c96f34df91dd1 (127.19.185.194:40567)
W20260926 02:54:06.994885 30399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:54:06.995169 30399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:54:06.995265 30399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:54:07.002669 30399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:54:07.002774 30399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:54:07.002882 30399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:54:07.012044 30399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:54:07.013309 30399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:54:07.015043 30399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:54:07.028237 30404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:07.028264 30405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:07.028563 30407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:54:07.029153 30399 server_base.cc:1034] running on GCE node
I20260926 02:54:07.029780 30399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:54:07.030710 30399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:54:07.031942 30399 hybrid_clock.cc:648] HybridClock initialized: now 1790391247031884 us; error 46 us; skew 500 ppm
I20260926 02:54:07.032279 30399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:54:07.034577 30399 webserver.cc:460] Webserver started at http://127.19.185.195:37131/ using document root <none> and password file <none>
I20260926 02:54:07.035434 30399 fs_manager.cc:359] Metadata directory not provided
I20260926 02:54:07.035557 30399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:54:07.035889 30399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:54:07.038853 30399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "57eb762ce27f4b0d850469efe032ec2b"
format_stamp: "Formatted at 2026-09-26 02:54:07 on dist-test-slave-ktng"
server_key: "d0dc4239a2fc5b834f57ef4c2d2803c8"
server_key_iv: "42c88e29bfc0f7913dd3741a7026092b"
server_key_version: "encryptionkey@0"
I20260926 02:54:07.039731 30399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "57eb762ce27f4b0d850469efe032ec2b"
format_stamp: "Formatted at 2026-09-26 02:54:07 on dist-test-slave-ktng"
server_key: "d0dc4239a2fc5b834f57ef4c2d2803c8"
server_key_iv: "42c88e29bfc0f7913dd3741a7026092b"
server_key_version: "encryptionkey@0"
I20260926 02:54:07.044872 30399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:54:07.048866 30413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:07.050444 30399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 02:54:07.050736 30399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "57eb762ce27f4b0d850469efe032ec2b"
format_stamp: "Formatted at 2026-09-26 02:54:07 on dist-test-slave-ktng"
server_key: "d0dc4239a2fc5b834f57ef4c2d2803c8"
server_key_iv: "42c88e29bfc0f7913dd3741a7026092b"
server_key_version: "encryptionkey@0"
I20260926 02:54:07.050992 30399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:54:07.056931 30399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:54:07.057790 30399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:54:07.058097 30399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:54:07.059046 30399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:54:07.060647 30399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:54:07.060773 30399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:07.060906 30399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:54:07.061004 30399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:07.094817 30399 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:35189
I20260926 02:54:07.094868 30525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:35189 every 8 connection(s)
I20260926 02:54:07.096246 30399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:54:07.103348 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 30399
I20260926 02:54:07.103555 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:54:07.104007 20199 external_mini_cluster.cc:1428] Setting key faf6681388d671a9657dc566070229e2
I20260926 02:54:07.104842 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:07.098947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:33451 (local address 127.19.185.254:37909)
0926 02:54:07.099169 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:07.099174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:07.100712 (+ 1538us) server_negotiation.cc:407] Connection header received
0926 02:54:07.102430 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:07.102436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:07.102512 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:07.102627 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:07.104038 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:07.104043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:07.104048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:54:07.104067 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:07.104139 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:54:07.104155 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:07.104278 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:07.104471 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 02:54:07.104541 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:54:07.105839 30528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:07.099543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:37909 (local address 127.19.185.195:33451)
0926 02:54:07.100520 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:07.100564 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:07.102154 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:07.102862 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:07.102871 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:07.102983 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:07.103741 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:54:07.103744 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:07.103783 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:54:07.103871 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:54:07.104254 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:54:07.104264 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:54:07.104272 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 02:54:07.104552 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 02:54:07.104825 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":262,"threads_started":1}
I20260926 02:54:07.107276 30526 heartbeater.cc:347] Connected to a master server at 127.19.185.254:37909
I20260926 02:54:07.107596 30526 heartbeater.cc:464] Registering TS with master...
I20260926 02:54:07.108188 30526 heartbeater.cc:511] Master 127.19.185.254:37909 requested a full tablet report, sending...
I20260926 02:54:07.109618 30078 ts_manager.cc:194] Registered new tserver with Master: 57eb762ce27f4b0d850469efe032ec2b (127.19.185.195:35189)
I20260926 02:54:07.119673 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 02:54:07.125866 30267 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:54:07.124148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.19.185.254:37909)
0926 02:54:07.124317 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:07.124322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:07.124875 (+ 553us) server_negotiation.cc:407] Connection header received
0926 02:54:07.125065 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:07.125071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:07.125175 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:07.125285 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:07.125616 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33676: BlockingRecv error: recv got EOF from 127.0.0.1:33676 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:54:07.126925 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 30138
I20260926 02:54:07.130420 30535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:07.127946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33794 (local address 127.19.185.193:37045)
0926 02:54:07.128510 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:07.128517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:07.128542 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:54:07.128599 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:07.128605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:07.128818 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:07.128943 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:07.129769 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:07.129775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:07.129787 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:54:07.129805 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:07.129847 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:54:07.129862 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:07.130046 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:07.130107 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 02:54:07.130184 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":277,"threads_started":1}
I20260926 02:54:07.131790 30260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:54:07.219841 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 30138
I20260926 02:54:07.236680 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 30269
I20260926 02:54:07.240545 30537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:07.237799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.19.185.194:40567)
0926 02:54:07.238416 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:07.238429 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:07.238455 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:54:07.238522 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:07.238529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:07.238749 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:07.238875 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:07.239835 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:07.239841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:07.239853 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:54:07.239871 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:07.239914 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:54:07.239928 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:07.240033 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:07.240225 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 02:54:07.240307 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":371,"threads_started":1}
I20260926 02:54:07.241773 30391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:54:07.322781 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 30269
I20260926 02:54:07.342157 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 30399
I20260926 02:54:07.346143 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:07.343189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.19.185.195:35189)
0926 02:54:07.343874 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:07.343881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:07.343912 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:54:07.343974 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:07.343980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:07.344204 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:07.344335 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:07.345368 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:07.345376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:07.345388 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:54:07.345403 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:07.345470 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:54:07.345490 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:07.345637 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:07.345720 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 02:54:07.345803 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":360,"threads_started":1}
I20260926 02:54:07.347615 30521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:54:07.429530 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 30399
I20260926 02:54:07.445569 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 30048
I20260926 02:54:07.446868 30109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:54:07.547994 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 30048
2026-09-26T02:54:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2354 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:54:07 dist-test-slave-ktng krb5kdc[30544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:54:07 dist-test-slave-ktng krb5kdc[30544](info): set up 2 sockets
Sep 26 02:54:07 dist-test-slave-ktng krb5kdc[30544](info): commencing operation
krb5kdc: starting...
W20260926 02:54:08.661068 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:54:08 dist-test-slave-ktng krb5kdc[30544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:54:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:54:08.898940 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:44223
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:40221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:44223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:54:09.231230 30557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:54:09.231604 30557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:54:09.231761 30557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:54:09.239035 30557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:54:09.239181 30557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:54:09.239308 30557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:54:09.239413 30557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:54:09.239487 30557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:54:09.249359 30557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:40221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:44223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:44223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:54:09.250797 30557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:54:09.252164 30557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:54:09.264226 30562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:09.264276 30563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:09.264351 30565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:54:09.265254 30557 server_base.cc:1034] running on GCE node
I20260926 02:54:09.266000 30557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:54:09.267473 30557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:54:09.268728 30557 hybrid_clock.cc:648] HybridClock initialized: now 1790391249268682 us; error 54 us; skew 500 ppm
I20260926 02:54:09.269078 30557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:54:09 dist-test-slave-ktng krb5kdc[30544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391249, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:54:09.277751 30557 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:54:09.279366 30557 webserver.cc:460] Webserver started at http://127.19.185.254:34265/ using document root <none> and password file <none>
I20260926 02:54:09.280064 30557 fs_manager.cc:359] Metadata directory not provided
I20260926 02:54:09.280221 30557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:54:09.280607 30557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:54:09.283696 30557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "206f0722a1074b2b86ac72b3328dce32"
format_stamp: "Formatted at 2026-09-26 02:54:09 on dist-test-slave-ktng"
server_key: "61bdebdf1de71d642135f5bf2a7dcfb8"
server_key_iv: "062984ad51b2feb32ae2a9a267acbbf2"
server_key_version: "encryptionkey@0"
I20260926 02:54:09.284451 30557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "206f0722a1074b2b86ac72b3328dce32"
format_stamp: "Formatted at 2026-09-26 02:54:09 on dist-test-slave-ktng"
server_key: "61bdebdf1de71d642135f5bf2a7dcfb8"
server_key_iv: "062984ad51b2feb32ae2a9a267acbbf2"
server_key_version: "encryptionkey@0"
I20260926 02:54:09.289665 30557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:54:09.293239 30572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:09.294557 30557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:54:09.294744 30557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "206f0722a1074b2b86ac72b3328dce32"
format_stamp: "Formatted at 2026-09-26 02:54:09 on dist-test-slave-ktng"
server_key: "61bdebdf1de71d642135f5bf2a7dcfb8"
server_key_iv: "062984ad51b2feb32ae2a9a267acbbf2"
server_key_version: "encryptionkey@0"
I20260926 02:54:09.294947 30557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:54:09.315469 30557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:54:09.319456 30557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:54:09.319754 30557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:54:09.338600 30557 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:44223
I20260926 02:54:09.338639 30623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:44223 every 8 connection(s)
I20260926 02:54:09.340180 30557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:54:09.342001 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 30557
I20260926 02:54:09.342249 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:54:09.343113 20199 external_mini_cluster.cc:1428] Setting key 4b97c1f537cd374e0b1fdf950057e592
I20260926 02:54:09.345382 30624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:54:09 dist-test-slave-ktng krb5kdc[30544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:54:09.359135 30624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32: Bootstrap starting.
I20260926 02:54:09.362172 30624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32: Neither blocks nor log segments found. Creating new log.
I20260926 02:54:09.363209 30624 log.cc:824] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32: Log is configured to *not* fsync() on all Append() calls
I20260926 02:54:09.366012 30624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32: No bootstrap required, opened a new log
I20260926 02:54:09.367939 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:09.345434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.19.185.254:44223)
0926 02:54:09.346334 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:09.346352 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:09.346394 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:54:09.348194 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:09.348218 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:09.348631 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:09.348986 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:09.350820 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:09.352441 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:09.354174 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:09.354905 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:09.355050 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:09.360042 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:09.360051 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:09.360065 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:54:09.360104 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:09.363559 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:09.364335 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:09.364344 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:09.364350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:09.364534 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:09.364951 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:09.364961 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:09.364965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:09.365530 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:09.365858 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:09.366339 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 02:54:09.366811 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":400,"threads_started":1}
I20260926 02:54:09.373793 30624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206f0722a1074b2b86ac72b3328dce32" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44223 } }
I20260926 02:54:09.374151 30624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:54:09.374259 30624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206f0722a1074b2b86ac72b3328dce32, State: Initialized, Role: FOLLOWER
I20260926 02:54:09.374794 30624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206f0722a1074b2b86ac72b3328dce32" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44223 } }
I20260926 02:54:09.375023 30624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:54:09.375161 30624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:54:09.375396 30624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:54:09.377897 30624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206f0722a1074b2b86ac72b3328dce32" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44223 } }
I20260926 02:54:09.378392 30624 leader_election.cc:302] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 206f0722a1074b2b86ac72b3328dce32; no voters:
I20260926 02:54:09.378954 30624 leader_election.cc:288] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:54:09.379112 30629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:54:09.380187 30629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [term 1 LEADER]: Becoming Leader. State: Replica: 206f0722a1074b2b86ac72b3328dce32, State: Running, Role: LEADER
I20260926 02:54:09.380694 30629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206f0722a1074b2b86ac72b3328dce32" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44223 } }
I20260926 02:54:09.381070 30624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:54:09.383913 30630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "206f0722a1074b2b86ac72b3328dce32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206f0722a1074b2b86ac72b3328dce32" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44223 } } }
I20260926 02:54:09.384264 30630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [sys.catalog]: This master's current role is: LEADER
I20260926 02:54:09.384699 30631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 206f0722a1074b2b86ac72b3328dce32. Latest consensus state: current_term: 1 leader_uuid: "206f0722a1074b2b86ac72b3328dce32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206f0722a1074b2b86ac72b3328dce32" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44223 } } }
I20260926 02:54:09.385035 30631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32 [sys.catalog]: This master's current role is: LEADER
I20260926 02:54:09.385296 30638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:54:09.391243 30638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:54:09.400238 30638 catalog_manager.cc:1347] Generated new cluster ID: 1e24acf216e549899e5d4243a689845e
I20260926 02:54:09.400367 30638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:54:09.416512 30638 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:54:09.418074 30638 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:54:09.427048 30638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32: Generated new TSK 0
I20260926 02:54:09.427984 30638 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:54:09.487896 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:44223
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:40221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:54:09.826467 30650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:54:09.826809 30650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:54:09.826983 30650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:54:09.835712 30650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:54:09.835856 30650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:54:09.836040 30650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:54:09.846122 30650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:40221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:44223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:54:09.847640 30650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:54:09.848975 30650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:54:09.862289 30656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:09.862360 30655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:09.862510 30658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:54:09.863279 30650 server_base.cc:1034] running on GCE node
I20260926 02:54:09.863916 30650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:54:09.864856 30650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:54:09.866057 30650 hybrid_clock.cc:648] HybridClock initialized: now 1790391249865960 us; error 86 us; skew 500 ppm
I20260926 02:54:09.866403 30650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:54:09 dist-test-slave-ktng krb5kdc[30544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391249, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:54:09.869746 30650 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:54:09.871335 30650 webserver.cc:460] Webserver started at http://127.19.185.193:42479/ using document root <none> and password file <none>
I20260926 02:54:09.872049 30650 fs_manager.cc:359] Metadata directory not provided
I20260926 02:54:09.872200 30650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:54:09.872494 30650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:54:09.875450 30650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "e526839b43464bfa885354fe21e253c0"
format_stamp: "Formatted at 2026-09-26 02:54:09 on dist-test-slave-ktng"
server_key: "c4cdef99082e21a22cfc43e751cd2990"
server_key_iv: "3d712b83e71ae554a2ba751056b3ddb7"
server_key_version: "encryptionkey@0"
I20260926 02:54:09.876241 30650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "e526839b43464bfa885354fe21e253c0"
format_stamp: "Formatted at 2026-09-26 02:54:09 on dist-test-slave-ktng"
server_key: "c4cdef99082e21a22cfc43e751cd2990"
server_key_iv: "3d712b83e71ae554a2ba751056b3ddb7"
server_key_version: "encryptionkey@0"
I20260926 02:54:09.881456 30650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 02:54:09.885386 30665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:09.886966 30650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:54:09.887100 30650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "e526839b43464bfa885354fe21e253c0"
format_stamp: "Formatted at 2026-09-26 02:54:09 on dist-test-slave-ktng"
server_key: "c4cdef99082e21a22cfc43e751cd2990"
server_key_iv: "3d712b83e71ae554a2ba751056b3ddb7"
server_key_version: "encryptionkey@0"
I20260926 02:54:09.887264 30650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:54:09.904667 30650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:54:09.909206 30650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:54:09.909526 30650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:54:09.910531 30650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:54:09.912053 30650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:54:09.912137 30650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:09.912230 30650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:54:09.912276 30650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:09.947070 30650 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:44339
I20260926 02:54:09.947118 30777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:44339 every 8 connection(s)
I20260926 02:54:09.948380 30650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:54:09.953415 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 30650
I20260926 02:54:09.953634 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:54:09.954409 20199 external_mini_cluster.cc:1428] Setting key eee7c5b322040b8806d669cd7be703ba
Sep 26 02:54:09 dist-test-slave-ktng krb5kdc[30544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391249, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:54:09.968266 30780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:09.950859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:40897 (local address 127.19.185.254:44223)
0926 02:54:09.951336 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:09.951343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:09.952292 (+ 949us) server_negotiation.cc:407] Connection header received
0926 02:54:09.954109 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:09.954116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:09.954210 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:09.954345 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:09.956213 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:09.956934 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:09.958525 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:09.958909 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:09.959038 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:09.962490 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:09.962495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:09.962498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:54:09.962529 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:09.965180 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:09.965925 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:09.965930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:09.965936 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:09.966059 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:09.966493 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:09.966497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:09.966500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:09.966758 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:09.966911 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:09.967673 (+ 762us) server_negotiation.cc:299] Negotiation successful
0926 02:54:09.967894 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":223,"threads_started":1}
I20260926 02:54:09.969240 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:09.951437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:44223 (local address 127.19.185.193:40897)
0926 02:54:09.952127 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:09.952171 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:09.953848 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:09.954580 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:09.954589 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:09.955219 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:09.956015 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:09.956040 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:09.957112 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:09.957116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:09.958352 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:09.958364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:09.959146 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:09.959153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:09.959450 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:09.960136 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:54:09.960162 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:09.962264 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:54:09.965339 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:54:09.965348 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:54:09.965372 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:54:09.965790 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:54:09.966186 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:54:09.966194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:54:09.966197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:54:09.966372 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:54:09.966886 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:54:09.966895 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:54:09.967242 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 02:54:09.967562 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 02:54:09.968099 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":263,"threads_started":1}
I20260926 02:54:09.970960 30778 heartbeater.cc:347] Connected to a master server at 127.19.185.254:44223
I20260926 02:54:09.971431 30778 heartbeater.cc:464] Registering TS with master...
I20260926 02:54:09.972215 30778 heartbeater.cc:511] Master 127.19.185.254:44223 requested a full tablet report, sending...
I20260926 02:54:09.974632 30589 ts_manager.cc:194] Registered new tserver with Master: e526839b43464bfa885354fe21e253c0 (127.19.185.193:44339)
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
I20260926 02:54:09.976850 30589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:40897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:54:10.007505 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:44223
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:40221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:54:10.332757 30784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:54:10.333058 30784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:54:10.333150 30784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:54:10.340626 30784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:54:10.340734 30784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:54:10.340912 30784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:54:10.350996 30784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:40221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:44223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:54:10.352242 30784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:54:10.353516 30784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:54:10.366608 30792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:10.366698 30789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:10.366662 30790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:54:10.367409 30784 server_base.cc:1034] running on GCE node
I20260926 02:54:10.367930 30784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:54:10.368857 30784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:54:10.370110 30784 hybrid_clock.cc:648] HybridClock initialized: now 1790391250370054 us; error 47 us; skew 500 ppm
I20260926 02:54:10.370374 30784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:54:10 dist-test-slave-ktng krb5kdc[30544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391250, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:54:10.373528 30784 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:54:10.374868 30784 webserver.cc:460] Webserver started at http://127.19.185.194:34503/ using document root <none> and password file <none>
I20260926 02:54:10.375566 30784 fs_manager.cc:359] Metadata directory not provided
I20260926 02:54:10.375703 30784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:54:10.376011 30784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:54:10.378854 30784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "2769a34d668b4b7ca894164f36065267"
format_stamp: "Formatted at 2026-09-26 02:54:10 on dist-test-slave-ktng"
server_key: "d14ec05543d943171d7adf100ff0d7a3"
server_key_iv: "a59ad8b4111938a388a4a0114eb2586d"
server_key_version: "encryptionkey@0"
I20260926 02:54:10.379665 30784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "2769a34d668b4b7ca894164f36065267"
format_stamp: "Formatted at 2026-09-26 02:54:10 on dist-test-slave-ktng"
server_key: "d14ec05543d943171d7adf100ff0d7a3"
server_key_iv: "a59ad8b4111938a388a4a0114eb2586d"
server_key_version: "encryptionkey@0"
I20260926 02:54:10.385010 30784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:54:10.388331 30799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:10.389680 30784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:54:10.389837 30784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "2769a34d668b4b7ca894164f36065267"
format_stamp: "Formatted at 2026-09-26 02:54:10 on dist-test-slave-ktng"
server_key: "d14ec05543d943171d7adf100ff0d7a3"
server_key_iv: "a59ad8b4111938a388a4a0114eb2586d"
server_key_version: "encryptionkey@0"
I20260926 02:54:10.390029 30784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:54:10.404651 30784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:54:10.408892 30784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:54:10.409178 30784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:54:10.410151 30784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:54:10.411569 30784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:54:10.411643 30784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:10.411798 30784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:54:10.411860 30784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:10.443248 30784 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:41293
I20260926 02:54:10.443284 30911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:41293 every 8 connection(s)
I20260926 02:54:10.444710 30784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:54:10.450235 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 30784
I20260926 02:54:10.450438 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:54:10.450857 20199 external_mini_cluster.cc:1428] Setting key fb64ea7f69f3693d3750f53a25dafd89
Sep 26 02:54:10 dist-test-slave-ktng krb5kdc[30544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391250, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:54:10.465549 30780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:10.447256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:33833 (local address 127.19.185.254:44223)
0926 02:54:10.447463 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:10.447468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:10.448598 (+ 1130us) server_negotiation.cc:407] Connection header received
0926 02:54:10.450443 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:10.450448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:10.450541 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:10.450662 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:10.452350 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:10.453210 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:10.454777 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:10.455211 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:10.455373 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:10.458731 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:10.458735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:10.458738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:54:10.458769 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:10.462325 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:10.463026 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:10.463035 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:10.463041 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:10.463207 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:10.463661 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:10.463668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:10.463674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:10.464036 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:10.464203 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:10.464825 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 02:54:10.465080 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 02:54:10.466187 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:10.447743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:44223 (local address 127.19.185.194:33833)
0926 02:54:10.448437 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:10.448481 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:10.450167 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:10.450890 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:10.450898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:10.451547 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:10.452179 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:10.452196 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:10.453379 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:10.453383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:10.454626 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:10.454639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:10.455452 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:10.455461 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:10.455700 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:10.456377 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:54:10.456401 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:10.458497 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:54:10.462463 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:54:10.462470 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:54:10.462483 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:54:10.462889 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:54:10.463367 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:54:10.463370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:54:10.463372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:54:10.463540 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:54:10.464155 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:54:10.464160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:54:10.464469 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 02:54:10.464691 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 02:54:10.465063 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":263,"threads_started":1}
I20260926 02:54:10.467727 30912 heartbeater.cc:347] Connected to a master server at 127.19.185.254:44223
I20260926 02:54:10.468058 30912 heartbeater.cc:464] Registering TS with master...
I20260926 02:54:10.468963 30912 heartbeater.cc:511] Master 127.19.185.254:44223 requested a full tablet report, sending...
I20260926 02:54:10.470604 30589 ts_manager.cc:194] Registered new tserver with Master: 2769a34d668b4b7ca894164f36065267 (127.19.185.194:41293)
I20260926 02:54:10.471463 30589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:33833
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:54:10.507017 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:44223
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:40221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:54:10.834651 30917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:54:10.834950 30917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:54:10.835112 30917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:54:10.843752 30917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:54:10.843858 30917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:54:10.844027 30917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:54:10.854102 30917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:40221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:44223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:54:10.855540 30917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:54:10.856841 30917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:54:10.870963 30925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:10.870913 30923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:10.871029 30922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:54:10.877170 30917 server_base.cc:1034] running on GCE node
I20260926 02:54:10.877821 30917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:54:10.878813 30917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:54:10.880038 30917 hybrid_clock.cc:648] HybridClock initialized: now 1790391250879979 us; error 56 us; skew 500 ppm
I20260926 02:54:10.880362 30917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:54:10 dist-test-slave-ktng krb5kdc[30544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391250, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:54:10.883646 30917 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:54:10.885300 30917 webserver.cc:460] Webserver started at http://127.19.185.195:40253/ using document root <none> and password file <none>
I20260926 02:54:10.885986 30917 fs_manager.cc:359] Metadata directory not provided
I20260926 02:54:10.886085 30917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:54:10.886327 30917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:54:10.889463 30917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "c0477ff7a5a9447d9108bceadcb01984"
format_stamp: "Formatted at 2026-09-26 02:54:10 on dist-test-slave-ktng"
server_key: "f207eabc221e55313197092525bfddc7"
server_key_iv: "c20c4d8c858c9ac34c57262f83bcd71c"
server_key_version: "encryptionkey@0"
I20260926 02:54:10.890228 30917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "c0477ff7a5a9447d9108bceadcb01984"
format_stamp: "Formatted at 2026-09-26 02:54:10 on dist-test-slave-ktng"
server_key: "f207eabc221e55313197092525bfddc7"
server_key_iv: "c20c4d8c858c9ac34c57262f83bcd71c"
server_key_version: "encryptionkey@0"
I20260926 02:54:10.895654 30917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:54:10.899067 30932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:10.900660 30917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:54:10.900882 30917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "c0477ff7a5a9447d9108bceadcb01984"
format_stamp: "Formatted at 2026-09-26 02:54:10 on dist-test-slave-ktng"
server_key: "f207eabc221e55313197092525bfddc7"
server_key_iv: "c20c4d8c858c9ac34c57262f83bcd71c"
server_key_version: "encryptionkey@0"
I20260926 02:54:10.901116 30917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:54:10.918344 30917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:54:10.922798 30917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:54:10.923141 30917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:54:10.924324 30917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:54:10.925848 30917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:54:10.925928 30917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:10.926213 30917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:54:10.926288 30917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:10.960234 30917 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:40315
I20260926 02:54:10.960273 31044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:40315 every 8 connection(s)
I20260926 02:54:10.961519 30917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:54:10.970130 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 30917
I20260926 02:54:10.970350 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:54:10.970778 20199 external_mini_cluster.cc:1428] Setting key d82dc09608347f1b1bbd230f0f95f7ed
Sep 26 02:54:10 dist-test-slave-ktng krb5kdc[30544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391250, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:54:10.980017 30778 heartbeater.cc:503] Master 127.19.185.254:44223 was elected leader, sending a full tablet report...
I20260926 02:54:10.982024 30780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:10.964083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:37105 (local address 127.19.185.254:44223)
0926 02:54:10.964279 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:10.964284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:10.965374 (+ 1090us) server_negotiation.cc:407] Connection header received
0926 02:54:10.967218 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:10.967223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:10.967378 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:10.967492 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:10.968954 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:10.969701 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:10.971588 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:10.972118 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:10.972248 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:10.976112 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:10.976117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:10.976120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:54:10.976155 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:10.978610 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:10.979390 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:10.979394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:10.979396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:10.979527 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:10.979936 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:10.979943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:10.979946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:10.980354 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:10.980507 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:10.981378 (+ 871us) server_negotiation.cc:299] Negotiation successful
0926 02:54:10.981681 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 02:54:10.983028 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:10.964509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:44223 (local address 127.19.185.195:37105)
0926 02:54:10.965212 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:10.965248 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:10.966933 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:10.967730 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:10.967738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:10.968231 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:54:10.968780 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:10.968797 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:10.969890 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:10.969895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:10.971412 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:10.971426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:10.972386 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:10.972403 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:10.972639 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:10.973418 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:54:10.973441 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:54:10.975922 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:54:10.978738 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:54:10.978746 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:54:10.978759 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:54:10.979198 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:54:10.979631 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:54:10.979635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:54:10.979637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:54:10.979814 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:54:10.980506 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:54:10.980515 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:54:10.980984 (+ 469us) client_negotiation.cc:769] Sending connection context
0926 02:54:10.981240 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 02:54:10.981822 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":255,"threads_started":1}
I20260926 02:54:10.984535 31045 heartbeater.cc:347] Connected to a master server at 127.19.185.254:44223
I20260926 02:54:10.984907 31045 heartbeater.cc:464] Registering TS with master...
I20260926 02:54:10.985821 31045 heartbeater.cc:511] Master 127.19.185.254:44223 requested a full tablet report, sending...
I20260926 02:54:10.987341 30589 ts_manager.cc:194] Registered new tserver with Master: c0477ff7a5a9447d9108bceadcb01984 (127.19.185.195:40315)
I20260926 02:54:10.988164 30589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:37105
I20260926 02:54:10.998963 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:54:11.015892 30780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.003641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.19.185.254:44223)
0926 02:54:11.003803 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:11.003807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:11.004342 (+ 535us) server_negotiation.cc:407] Connection header received
0926 02:54:11.004542 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:11.004547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:11.004618 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:11.004749 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:11.006246 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.007041 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.008238 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.008623 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.008743 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.010892 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:11.010896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:11.010900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:54:11.010931 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:11.013605 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:11.014207 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:11.014211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:11.014214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:11.014339 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:11.014648 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:11.014651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:11.014652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:11.014974 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:11.015143 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:11.015482 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 02:54:11.015639 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:54:11.019985 30589 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:54:11.026684 30589 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:54:11.063730 31058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.039286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.19.185.195:40315)
0926 02:54:11.055434 (+ 16148us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:11.055444 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:11.055480 (+ 36us) server_negotiation.cc:407] Connection header received
0926 02:54:11.055582 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:11.055590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:11.055905 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:11.056151 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:54:11.057119 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.058605 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.061121 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.062474 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.062665 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.062850 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:11.063186 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 02:54:11.063381 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15920,"thread_start_us":15769,"threads_started":1}
I20260926 02:54:11.063779 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.039073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:40315 (local address 127.0.0.1:34752)
0926 02:54:11.054327 (+ 15254us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:11.054373 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:11.054568 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:11.056402 (+ 1834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:11.056412 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:11.056452 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:54:11.056903 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:11.056918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.058814 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.058820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:11.060918 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:11.060936 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.062711 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.062720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:11.062919 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.062975 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 02:54:11.063113 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 02:54:11.063333 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14985,"mutex_wait_us":39,"spinlock_wait_cycles":1792,"thread_start_us":14964,"threads_started":1}
I20260926 02:54:11.064975 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.039439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:44339 (local address 127.0.0.1:45372)
0926 02:54:11.054570 (+ 15131us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:11.054594 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:11.054762 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:11.057499 (+ 2737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:11.057505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:11.057535 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:54:11.058003 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:11.058016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.059731 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.059743 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:11.062306 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:11.062320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.064198 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.064208 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:11.064379 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.064413 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:54:11.064480 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 02:54:11.064597 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14959,"mutex_wait_us":71,"thread_start_us":14437,"threads_started":1}
I20260926 02:54:11.065975 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.039969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.19.185.193:44339)
0926 02:54:11.056923 (+ 16954us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:11.056932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:11.056962 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:54:11.057062 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:11.057069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:11.057255 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:11.058174 (+ 919us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:54:11.058352 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.059588 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.062481 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.063956 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.065281 (+ 1325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.065448 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:11.065518 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 02:54:11.065682 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16769,"thread_start_us":16597,"threads_started":1}
I20260926 02:54:11.065986 31061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.040794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:41293 (local address 127.0.0.1:44870)
0926 02:54:11.056465 (+ 15671us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:11.056489 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:11.056646 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:11.057346 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:11.057351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:11.057378 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:54:11.057872 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:11.057887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.059812 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.059830 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:11.062664 (+ 2834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:11.062677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.065160 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.065178 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:11.065362 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.065396 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:54:11.065498 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 02:54:11.065624 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15496,"mutex_wait_us":113,"thread_start_us":15329,"threads_started":1}
I20260926 02:54:11.066032 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.041048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44870 (local address 127.19.185.194:41293)
0926 02:54:11.055930 (+ 14882us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:11.055942 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:11.056780 (+ 838us) server_negotiation.cc:407] Connection header received
0926 02:54:11.056892 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:11.056897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:11.057130 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:11.057349 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:54:11.058107 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.059686 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.062842 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.064607 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.064768 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.064921 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:11.065548 (+ 627us) server_negotiation.cc:299] Negotiation successful
0926 02:54:11.065714 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14707,"thread_start_us":14558,"threads_started":1}
I20260926 02:54:11.068120 30980 tablet_service.cc:1467] Processing CreateTablet for tablet 8685cded998c487687714394783aa150 (DEFAULT_TABLE table=test-table [id=3adfcd6287bc49888dbedeeb875fb501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:54:11.069720 30980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8685cded998c487687714394783aa150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:54:11.069389 30713 tablet_service.cc:1467] Processing CreateTablet for tablet 8685cded998c487687714394783aa150 (DEFAULT_TABLE table=test-table [id=3adfcd6287bc49888dbedeeb875fb501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:54:11.070518 30713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8685cded998c487687714394783aa150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:54:11.070818 30847 tablet_service.cc:1467] Processing CreateTablet for tablet 8685cded998c487687714394783aa150 (DEFAULT_TABLE table=test-table [id=3adfcd6287bc49888dbedeeb875fb501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:54:11.072127 30847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8685cded998c487687714394783aa150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:54:11.081557 31063 tablet_bootstrap.cc:492] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0: Bootstrap starting.
I20260926 02:54:11.084753 31064 tablet_bootstrap.cc:492] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984: Bootstrap starting.
I20260926 02:54:11.085318 31063 tablet_bootstrap.cc:654] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0: Neither blocks nor log segments found. Creating new log.
I20260926 02:54:11.086161 31065 tablet_bootstrap.cc:492] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267: Bootstrap starting.
I20260926 02:54:11.086989 31063 log.cc:824] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:54:11.088235 31064 tablet_bootstrap.cc:654] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984: Neither blocks nor log segments found. Creating new log.
I20260926 02:54:11.089388 31064 log.cc:824] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984: Log is configured to *not* fsync() on all Append() calls
I20260926 02:54:11.089800 31065 tablet_bootstrap.cc:654] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267: Neither blocks nor log segments found. Creating new log.
I20260926 02:54:11.091048 31065 log.cc:824] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267: Log is configured to *not* fsync() on all Append() calls
I20260926 02:54:11.091651 31063 tablet_bootstrap.cc:492] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0: No bootstrap required, opened a new log
I20260926 02:54:11.091926 31063 ts_tablet_manager.cc:1397] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260926 02:54:11.092483 31064 tablet_bootstrap.cc:492] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984: No bootstrap required, opened a new log
I20260926 02:54:11.092736 31064 ts_tablet_manager.cc:1397] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 02:54:11.093662 31065 tablet_bootstrap.cc:492] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267: No bootstrap required, opened a new log
I20260926 02:54:11.093918 31065 ts_tablet_manager.cc:1397] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 02:54:11.101071 31065 raft_consensus.cc:348] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0477ff7a5a9447d9108bceadcb01984" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40315 } } peers { permanent_uuid: "e526839b43464bfa885354fe21e253c0" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44339 } } peers { permanent_uuid: "2769a34d668b4b7ca894164f36065267" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41293 } }
I20260926 02:54:11.101532 31065 raft_consensus.cc:374] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:54:11.101634 31065 raft_consensus.cc:729] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2769a34d668b4b7ca894164f36065267, State: Initialized, Role: FOLLOWER
I20260926 02:54:11.102341 31065 consensus_queue.cc:260] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0477ff7a5a9447d9108bceadcb01984" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40315 } } peers { permanent_uuid: "e526839b43464bfa885354fe21e253c0" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44339 } } peers { permanent_uuid: "2769a34d668b4b7ca894164f36065267" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41293 } }
I20260926 02:54:11.103452 30912 heartbeater.cc:503] Master 127.19.185.254:44223 was elected leader, sending a full tablet report...
I20260926 02:54:11.103259 31064 raft_consensus.cc:348] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0477ff7a5a9447d9108bceadcb01984" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40315 } } peers { permanent_uuid: "e526839b43464bfa885354fe21e253c0" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44339 } } peers { permanent_uuid: "2769a34d668b4b7ca894164f36065267" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41293 } }
I20260926 02:54:11.103262 31063 raft_consensus.cc:348] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0477ff7a5a9447d9108bceadcb01984" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40315 } } peers { permanent_uuid: "e526839b43464bfa885354fe21e253c0" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44339 } } peers { permanent_uuid: "2769a34d668b4b7ca894164f36065267" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41293 } }
I20260926 02:54:11.103674 31063 raft_consensus.cc:374] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:54:11.103686 31064 raft_consensus.cc:374] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:54:11.103768 31063 raft_consensus.cc:729] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e526839b43464bfa885354fe21e253c0, State: Initialized, Role: FOLLOWER
I20260926 02:54:11.103768 31064 raft_consensus.cc:729] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0477ff7a5a9447d9108bceadcb01984, State: Initialized, Role: FOLLOWER
I20260926 02:54:11.104111 31065 ts_tablet_manager.cc:1428] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260926 02:54:11.104434 31064 consensus_queue.cc:260] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0477ff7a5a9447d9108bceadcb01984" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40315 } } peers { permanent_uuid: "e526839b43464bfa885354fe21e253c0" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44339 } } peers { permanent_uuid: "2769a34d668b4b7ca894164f36065267" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41293 } }
I20260926 02:54:11.104434 31063 consensus_queue.cc:260] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0477ff7a5a9447d9108bceadcb01984" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40315 } } peers { permanent_uuid: "e526839b43464bfa885354fe21e253c0" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44339 } } peers { permanent_uuid: "2769a34d668b4b7ca894164f36065267" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41293 } }
I20260926 02:54:11.105556 31045 heartbeater.cc:503] Master 127.19.185.254:44223 was elected leader, sending a full tablet report...
I20260926 02:54:11.106155 31064 ts_tablet_manager.cc:1428] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260926 02:54:11.108369 31063 ts_tablet_manager.cc:1428] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260926 02:54:11.123416 31071 raft_consensus.cc:482] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:54:11.123719 31071 raft_consensus.cc:504] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0477ff7a5a9447d9108bceadcb01984" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40315 } } peers { permanent_uuid: "e526839b43464bfa885354fe21e253c0" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44339 } } peers { permanent_uuid: "2769a34d668b4b7ca894164f36065267" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41293 } }
I20260926 02:54:11.125365 31071 leader_election.cc:288] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e526839b43464bfa885354fe21e253c0 (127.19.185.193:44339), 2769a34d668b4b7ca894164f36065267 (127.19.185.194:41293)
I20260926 02:54:11.131897 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.125830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:42581 (local address 127.19.185.194:41293)
0926 02:54:11.126026 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:11.126033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:11.126207 (+ 174us) server_negotiation.cc:407] Connection header received
0926 02:54:11.126436 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:11.126439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:11.126495 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:11.126587 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:54:11.127015 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.128027 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.130007 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.130967 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.131081 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.131130 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:11.131562 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 02:54:11.131651 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:54:11.132037 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.125820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:41293 (local address 127.19.185.195:42581)
0926 02:54:11.126106 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:11.126124 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:11.126285 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:11.126611 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:11.126616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:11.126642 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:54:11.126868 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:11.126881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.128170 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.128177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:11.129852 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:11.129865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.131135 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.131142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:11.131372 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.131417 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 02:54:11.131545 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 02:54:11.131689 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260926 02:54:11.132699 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.125832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:43705 (local address 127.19.185.193:44339)
0926 02:54:11.126031 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:11.126038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:11.126790 (+ 752us) server_negotiation.cc:407] Connection header received
0926 02:54:11.126908 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:11.126916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:11.127000 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:11.127073 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:54:11.127692 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.128725 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.130533 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.131707 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.131867 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.131938 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:11.132360 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 02:54:11.132475 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:54:11.132793 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.125818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:44339 (local address 127.19.185.195:43705)
0926 02:54:11.126688 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 02:54:11.126705 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:54:11.126805 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:54:11.127117 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:54:11.127124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:54:11.127148 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:54:11.127523 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:11.127535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.128876 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.128880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:11.130366 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:54:11.130381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.131924 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.131932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:54:11.132127 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.132165 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:54:11.132301 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 02:54:11.132423 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":91,"thread_start_us":471,"threads_started":1}
I20260926 02:54:11.133208 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8685cded998c487687714394783aa150" candidate_uuid: "c0477ff7a5a9447d9108bceadcb01984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2769a34d668b4b7ca894164f36065267" is_pre_election: true
I20260926 02:54:11.133726 30867 raft_consensus.cc:2454] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0477ff7a5a9447d9108bceadcb01984 in term 0.
I20260926 02:54:11.133801 30733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8685cded998c487687714394783aa150" candidate_uuid: "c0477ff7a5a9447d9108bceadcb01984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e526839b43464bfa885354fe21e253c0" is_pre_election: true
I20260926 02:54:11.134332 30733 raft_consensus.cc:2454] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0477ff7a5a9447d9108bceadcb01984 in term 0.
I20260926 02:54:11.134438 30936 leader_election.cc:302] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2769a34d668b4b7ca894164f36065267, c0477ff7a5a9447d9108bceadcb01984; no voters:
I20260926 02:54:11.134920 31071 raft_consensus.cc:2781] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:54:11.135035 31071 raft_consensus.cc:482] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:54:11.135178 31071 raft_consensus.cc:3037] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:54:11.137588 31071 raft_consensus.cc:504] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0477ff7a5a9447d9108bceadcb01984" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40315 } } peers { permanent_uuid: "e526839b43464bfa885354fe21e253c0" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44339 } } peers { permanent_uuid: "2769a34d668b4b7ca894164f36065267" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41293 } }
I20260926 02:54:11.138175 31071 leader_election.cc:288] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [CANDIDATE]: Term 1 election: Requested vote from peers e526839b43464bfa885354fe21e253c0 (127.19.185.193:44339), 2769a34d668b4b7ca894164f36065267 (127.19.185.194:41293)
I20260926 02:54:11.138585 30733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8685cded998c487687714394783aa150" candidate_uuid: "c0477ff7a5a9447d9108bceadcb01984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e526839b43464bfa885354fe21e253c0"
I20260926 02:54:11.138599 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8685cded998c487687714394783aa150" candidate_uuid: "c0477ff7a5a9447d9108bceadcb01984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2769a34d668b4b7ca894164f36065267"
I20260926 02:54:11.138777 30733 raft_consensus.cc:3037] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:54:11.138793 30867 raft_consensus.cc:3037] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:54:11.140949 30733 raft_consensus.cc:2454] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0477ff7a5a9447d9108bceadcb01984 in term 1.
I20260926 02:54:11.140973 30867 raft_consensus.cc:2454] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0477ff7a5a9447d9108bceadcb01984 in term 1.
I20260926 02:54:11.141348 30934 leader_election.cc:302] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0477ff7a5a9447d9108bceadcb01984, e526839b43464bfa885354fe21e253c0; no voters:
I20260926 02:54:11.141683 31071 raft_consensus.cc:2781] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:54:11.142128 31071 raft_consensus.cc:686] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [term 1 LEADER]: Becoming Leader. State: Replica: c0477ff7a5a9447d9108bceadcb01984, State: Running, Role: LEADER
I20260926 02:54:11.142585 31071 consensus_queue.cc:237] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0477ff7a5a9447d9108bceadcb01984" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40315 } } peers { permanent_uuid: "e526839b43464bfa885354fe21e253c0" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44339 } } peers { permanent_uuid: "2769a34d668b4b7ca894164f36065267" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41293 } }
I20260926 02:54:11.147631 30587 catalog_manager.cc:5495] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0477ff7a5a9447d9108bceadcb01984 (127.19.185.195). New cstate: current_term: 1 leader_uuid: "c0477ff7a5a9447d9108bceadcb01984" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0477ff7a5a9447d9108bceadcb01984" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e526839b43464bfa885354fe21e253c0" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2769a34d668b4b7ca894164f36065267" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41293 } health_report { overall_health: UNKNOWN } } }
I20260926 02:54:11.178599 31058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.171677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.19.185.195:40315)
0926 02:54:11.171860 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:11.171865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:11.172324 (+ 459us) server_negotiation.cc:407] Connection header received
0926 02:54:11.172583 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:11.172588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:11.172670 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:11.172824 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:54:11.173360 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.174156 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.176104 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.176512 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.176654 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.177259 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:54:11.177999 (+ 740us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:54:11.178143 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:11.178226 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 02:54:11.178304 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 02:54:11.188102 30733 raft_consensus.cc:1261] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0 [term 1 FOLLOWER]: Refusing update from remote peer c0477ff7a5a9447d9108bceadcb01984: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:54:11.188143 30867 raft_consensus.cc:1261] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267 [term 1 FOLLOWER]: Refusing update from remote peer c0477ff7a5a9447d9108bceadcb01984: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:54:11.189167 31073 consensus_queue.cc:1035] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2769a34d668b4b7ca894164f36065267" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:54:11.189996 31073 consensus_queue.cc:1035] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e526839b43464bfa885354fe21e253c0" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 44339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 02:54:11.196647 30913 tablet.cc:2369] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:54:11.200613 30779 tablet.cc:2369] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:54:11.203145 31076 mvcc.cc:205] Tried to move back new op lower bound from 7333442564854640640 to 7333442564684161024. Current Snapshot: MvccSnapshot[applied={T|T < 7333442564854640640}]
I20260926 02:54:11.209942 31078 mvcc.cc:205] Tried to move back new op lower bound from 7333442564854640640 to 7333442564684161024. Current Snapshot: MvccSnapshot[applied={T|T < 7333442564854640640}]
I20260926 02:54:11.221529 30587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:54:11.221845 30587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:54:11.225716 30587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 206f0722a1074b2b86ac72b3328dce32: Sending DeleteTablet for 3 replicas of tablet 8685cded998c487687714394783aa150
I20260926 02:54:11.226981 30980 tablet_service.cc:1514] Processing DeleteTablet for tablet 8685cded998c487687714394783aa150 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:54:11 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:34752
I20260926 02:54:11.226989 30847 tablet_service.cc:1514] Processing DeleteTablet for tablet 8685cded998c487687714394783aa150 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:54:11 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:44870
I20260926 02:54:11.226989 30713 tablet_service.cc:1514] Processing DeleteTablet for tablet 8685cded998c487687714394783aa150 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:54:11 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:45372
I20260926 02:54:11.232177 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 30650
I20260926 02:54:11.239270 31087 tablet_replica.cc:330] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0: stopping tablet replica
I20260926 02:54:11.239751 31087 raft_consensus.cc:2229] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:54:11.240825 31086 tablet_replica.cc:330] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267: stopping tablet replica
I20260926 02:54:11.241225 31086 raft_consensus.cc:2229] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:54:11.241719 31086 raft_consensus.cc:2258] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:54:11.242142 31087 raft_consensus.cc:2258] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:54:11.244084 31085 tablet_replica.cc:330] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984: stopping tablet replica
I20260926 02:54:11.244207 31086 ts_tablet_manager.cc:1905] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:54:11.244460 31085 raft_consensus.cc:2229] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:54:11.245201 31085 raft_consensus.cc:2258] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:54:11.247385 31085 ts_tablet_manager.cc:1905] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:54:11.249018 31087 ts_tablet_manager.cc:1905] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 02:54:11 dist-test-slave-ktng krb5kdc[30544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:54:11.255169 31085 ts_tablet_manager.cc:1918] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:54:11.255375 31085 log.cc:1196] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/8685cded998c487687714394783aa150
I20260926 02:54:11.255935 31085 ts_tablet_manager.cc:1939] T 8685cded998c487687714394783aa150 P c0477ff7a5a9447d9108bceadcb01984: Deleting consensus metadata
I20260926 02:54:11.256198 31087 ts_tablet_manager.cc:1918] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:54:11.256345 31087 log.cc:1196] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/8685cded998c487687714394783aa150
I20260926 02:54:11.256855 31087 ts_tablet_manager.cc:1939] T 8685cded998c487687714394783aa150 P e526839b43464bfa885354fe21e253c0: Deleting consensus metadata
I20260926 02:54:11.257772 30575 catalog_manager.cc:4841] TS c0477ff7a5a9447d9108bceadcb01984 (127.19.185.195:40315): tablet 8685cded998c487687714394783aa150 (table test-table [id=3adfcd6287bc49888dbedeeb875fb501]) successfully deleted
I20260926 02:54:11.258589 30574 catalog_manager.cc:4841] TS e526839b43464bfa885354fe21e253c0 (127.19.185.193:44339): tablet 8685cded998c487687714394783aa150 (table test-table [id=3adfcd6287bc49888dbedeeb875fb501]) successfully deleted
I20260926 02:54:11.260885 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.233852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45386 (local address 127.19.185.193:44339)
0926 02:54:11.234097 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:11.234107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:11.240078 (+ 5971us) server_negotiation.cc:407] Connection header received
0926 02:54:11.240404 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:11.240413 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:11.240507 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:11.240592 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:11.243644 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.244731 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.247060 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.247682 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.247783 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.253157 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:11.253176 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:11.253195 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:54:11.253246 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:11.257289 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:11.258488 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:11.258497 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:11.258500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:11.258655 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:11.259158 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:11.259163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:11.259167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:11.259610 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:11.259810 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:11.260281 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 02:54:11.260498 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:54:11.261569 30773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:54:11.263114 31086 ts_tablet_manager.cc:1918] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:54:11.263386 31086 log.cc:1196] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/8685cded998c487687714394783aa150
I20260926 02:54:11.264308 31086 ts_tablet_manager.cc:1939] T 8685cded998c487687714394783aa150 P 2769a34d668b4b7ca894164f36065267: Deleting consensus metadata
I20260926 02:54:11.266902 30576 catalog_manager.cc:4841] TS 2769a34d668b4b7ca894164f36065267 (127.19.185.194:41293): tablet 8685cded998c487687714394783aa150 (table test-table [id=3adfcd6287bc49888dbedeeb875fb501]) successfully deleted
I20260926 02:54:11.363643 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 30650
I20260926 02:54:11.381348 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 30784
Sep 26 02:54:11 dist-test-slave-ktng krb5kdc[30544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:54:11.396852 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.382168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44880 (local address 127.19.185.194:41293)
0926 02:54:11.382347 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:11.382353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:11.382397 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:54:11.382677 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:11.382682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:11.382748 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:11.382864 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:11.384330 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.385067 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.386273 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.386711 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.386850 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.390811 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:11.390826 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:11.390842 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:54:11.390875 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:11.393897 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:11.394676 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:11.394691 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:11.394694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:11.394812 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:11.395204 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:11.395210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:11.395213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:11.395662 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:11.395959 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:11.396309 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 02:54:11.396519 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:54:11.397467 30907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:54:11.508461 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 30784
I20260926 02:54:11.525892 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 30917
Sep 26 02:54:11 dist-test-slave-ktng krb5kdc[30544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:54:11.543473 31058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:11.526867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34770 (local address 127.19.185.195:40315)
0926 02:54:11.527210 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:11.527217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:11.527241 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:54:11.527409 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:11.527416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:11.527497 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:11.527627 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:11.528940 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.530199 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.531539 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:11.532243 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:11.532380 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:11.537141 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:11.537153 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:11.537169 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:54:11.537211 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:11.540390 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:11.541278 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:11.541285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:11.541288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:11.541453 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:11.541869 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:11.541872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:11.541874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:11.542217 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:11.542445 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:11.542870 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 02:54:11.543033 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:54:11.544329 31040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:54:11.667423 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 30917
I20260926 02:54:11.687544 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 30557
I20260926 02:54:11.688710 30619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:54:11.796567 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 30557
2026-09-26T02:54:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4250 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:54:11 dist-test-slave-ktng krb5kdc[31095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:54:11 dist-test-slave-ktng krb5kdc[31095](info): set up 2 sockets
Sep 26 02:54:11 dist-test-slave-ktng krb5kdc[31095](info): commencing operation
krb5kdc: starting...
W20260926 02:54:12.908996 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:54:13 dist-test-slave-ktng krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:54:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.185.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.185.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/rangeradmin_127.19.185.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/rangeradmin_127.19.185.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.185.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.185.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/rangerlookup_127.19.185.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/rangerlookup_127.19.185.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.185.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.185.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/HTTP_127.19.185.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/HTTP_127.19.185.212@KRBTEST.COM.keytab.
I20260926 02:54:13.246307 20199 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAXkdvX/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/postgres -l logfile start
2026-09-26 02:54:16.123 UTC [31125] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 02:54:16.123 UTC [31125] LOG: listening on IPv4 address "127.19.185.212", port 52237
2026-09-26 02:54:16.127 UTC [31125] LOG: listening on Unix socket "/tmp/.s.PGSQL.52237"
2026-09-26 02:54:16.150 UTC [31128] LOG: database system was shut down at 2026-09-26 02:54:14 UTC
2026-09-26 02:54:16.154 UTC [31125] LOG: database system is ready to accept connections
I20260926 02:54:17.157371 20199 mini_postgres.cc:96] Postgres bound to 52237
2026-09-26 02:54:17.182 UTC [31136] FATAL: database "slave" does not exist
127.19.185.212:52237 - accepting connections
I20260926 02:54:17.183632 20199 mini_ranger.cc:161] Starting Ranger...
I20260926 02:54:17.211254 20199 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 02:54:17.759385 20199 mini_ranger.cc:87] Created ranger Postgres database
I20260926 02:54:17.759526 20199 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-admin
2026-09-26 02:54:18,277 [I] DB FLAVOR :POSTGRES
2026-09-26 02:54:18,277 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:54:18,277 [I] Checking connection..
2026-09-26 02:54:18,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 02:54:18,558 [I] Checking connection passed.
2026-09-26 02:54:18,558 [I] --------- Verifying version history table ---------
2026-09-26 02:54:18,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:54:18,848 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:54:18,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:54:19,120 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:54:19,120 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 02:54:19,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 02:54:19.388 UTC [31200] WARNING: there is no transaction in progress
2026-09-26 02:54:19,395 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 02:54:19,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:54:19,659 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 02:54:19,660 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:54:19,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 02:54:19,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 02:54:20,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ktng.c.gcp-upstream.internal','N') ;"
2026-09-26 02:54:20,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:54:20,669 [I] Table x_portal_user does not exist in database ranger
2026-09-26 02:54:20,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:54:20,931 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 02:54:20,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:54:21,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:54:21,446 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 02:54:21,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 02:54:22.482 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.499 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.518 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.535 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.552 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.571 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.637 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.645 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.662 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.674 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.685 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.700 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.711 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.718 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.725 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:22.733 UTC [31308] WARNING: there is no transaction in progress
2026-09-26 02:54:23,460 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 02:54:23,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 02:54:23,746 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 02:54:23,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:54:24,005 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 02:54:24,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:54:24,260 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 02:54:24,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:54:24,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:54:24,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ktng.c.gcp-upstream.internal';"
2026-09-26 02:54:25,009 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 02:54:25,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 02:54:25,263 [I] DB_PATCHES have already been applied
I20260926 02:54:25.269759 20199 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAXkdvX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 02:54:25.269855 20199 mini_ranger.cc:193] Using host: 127.19.185.212
I20260926 02:54:25.285681 20199 mini_ranger.cc:227] Ranger admin URL: http://127.19.185.212:60985
Sep 26, 2026 2:54:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:54:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:54:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:54:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:54:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:54:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:54:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 02:54:26.418926 20199 mini_ranger.cc:160] Time spent starting Ranger: real 9.235s user 0.000s sys 0.040s
Sep 26, 2026 2:54:46 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:54:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 2:54:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:54:46 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:54:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:54:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:54:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:54:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:54:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:54:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:54:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:54:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:54:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:54:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:54:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:54:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:54:46 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:54:47 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 02:54:48.558748 20199 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.185.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.185.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/rangerkms_127.19.185.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/rangerkms_127.19.185.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.185.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.185.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.185.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/HTTP_127.19.185.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.185.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/HTTP_127.19.185.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 02:54:48 dist-test-slave-ktng krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391288, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 02:54:48.725181 20199 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 02:54:48.755115 20199 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 02:54:49.703583 20199 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 02:54:49.703874 20199 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-kms
I20260926 02:54:49.704021 20199 mini_ranger_kms.cc:227] Using postgres at 127.19.185.212:52237
2026-09-26 02:54:50,564 [I] DB FLAVOR :POSTGRES
2026-09-26 02:54:50,564 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:54:50,564 [I] Checking connection
2026-09-26 02:54:50,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 02:54:50,800 [I] connection success
2026-09-26 02:54:50,801 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 02:54:50,801 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 02:54:50,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 02:54:51,069 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 02:54:51,070 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:54:51,070 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 02:54:51,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:52237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 02:54:51,337 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 02:54:51.832957 20199 mini_ranger_kms.cc:310] Created kms service
I20260926 02:54:52.050285 20199 mini_ranger_kms.cc:326] Created kudu user
I20260926 02:54:52.097189 20199 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 02:54:52 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391288, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:54:52.534289 20199 mini_ranger_kms.cc:384] Added ranger policy
I20260926 02:54:52.534631 20199 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 02:54:52.534690 20199 mini_ranger_kms.cc:240] Using host: 127.19.185.212
I20260926 02:54:52.561062 20199 mini_ranger_kms.cc:276] Ranger KMS PID: 31489
I20260926 02:54:52.561257 20199 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.185.212:53435
Sep 26, 2026 2:54:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:54:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:54:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:54:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:54:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:54:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 02:54:54 dist-test-slave-ktng krb5kdc[31095](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790391294, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.185.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:54:54.914985 20199 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.190s user 0.106s sys 0.426s
I20260926 02:54:54.915261 20199 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 02:54:54.915537 20199 mini_ranger_kms.cc:401] 127.19.185.212:53435/kms/v1/keys
Sep 26 02:54:55 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391294, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.185.212@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
Sep 26 02:54:55 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391294, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.185.212@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
Sep 26 02:54:57 dist-test-slave-ktng krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:54:57.979538 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:37559
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:37559
--ranger_config_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.185.212:53435/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:54:58.330420 31539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:54:58.330715 31539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:54:58.330871 31539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:54:58.338222 31539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:54:58.338330 31539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:54:58.338454 31539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:54:58.338544 31539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:54:58.338611 31539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:54:58.348953 31539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.185.212:53435/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:37559
--ranger_config_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:37559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:54:58.350234 31539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:54:58.351616 31539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:54:58.367129 31545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:58.367583 31544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:58.367705 31547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:54:58.367867 31539 server_base.cc:1034] running on GCE node
I20260926 02:54:58.368788 31539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:54:58.370273 31539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:54:58.371541 31539 hybrid_clock.cc:648] HybridClock initialized: now 1790391298371450 us; error 39 us; skew 500 ppm
I20260926 02:54:58.371996 31539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:54:58 dist-test-slave-ktng krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391298, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:54:58.382036 31539 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:54:58.383968 31539 webserver.cc:460] Webserver started at http://127.19.185.254:42227/ using document root <none> and password file <none>
I20260926 02:54:58.384785 31539 fs_manager.cc:359] Metadata directory not provided
I20260926 02:54:58.385003 31539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:54:58.385380 31539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:54:58 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391298, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:54:58.450227 31539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "0dcdbde380464052a1907d2c29fff760"
format_stamp: "Formatted at 2026-09-26 02:54:58 on dist-test-slave-ktng"
server_key: "e94bdc3fa4cf05c1530edc7c67fe3c6c"
server_key_iv: "ee56a968e617597645a4da800a385af2"
server_key_version: "kuduclusterkey@0"
I20260926 02:54:58.451077 31539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "0dcdbde380464052a1907d2c29fff760"
format_stamp: "Formatted at 2026-09-26 02:54:58 on dist-test-slave-ktng"
server_key: "e94bdc3fa4cf05c1530edc7c67fe3c6c"
server_key_iv: "ee56a968e617597645a4da800a385af2"
server_key_version: "kuduclusterkey@0"
I20260926 02:54:58.457319 31539 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260926 02:54:58.479555 31554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:54:58.481356 31539 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 02:54:58.481594 31539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "0dcdbde380464052a1907d2c29fff760"
format_stamp: "Formatted at 2026-09-26 02:54:58 on dist-test-slave-ktng"
server_key: "e94bdc3fa4cf05c1530edc7c67fe3c6c"
server_key_iv: "ee56a968e617597645a4da800a385af2"
server_key_version: "kuduclusterkey@0"
I20260926 02:54:58.481900 31539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:54:58.498332 31539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:54:58.502502 31539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:54:58.502825 31539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:54:58.522759 31539 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:37559
I20260926 02:54:58.522795 31605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:37559 every 8 connection(s)
I20260926 02:54:58.524210 31539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:54:58.528123 31606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:54:58.531744 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 31539
I20260926 02:54:58.531949 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:54:58.532636 20199 external_mini_cluster.cc:1428] Setting key c361f6158ee52feb7924f6564dd41646
I20260926 02:54:58.539140 31606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760: Bootstrap starting.
I20260926 02:54:58.542712 31606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760: Neither blocks nor log segments found. Creating new log.
I20260926 02:54:58.543897 31606 log.cc:824] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760: Log is configured to *not* fsync() on all Append() calls
I20260926 02:54:58.546640 31606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760: No bootstrap required, opened a new log
Sep 26 02:54:58 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:54:58.556382 31606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcdbde380464052a1907d2c29fff760" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37559 } }
I20260926 02:54:58.556833 31606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:54:58.556854 31609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:54:58.534638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.19.185.254:37559)
0926 02:54:58.535456 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0926 02:54:58.535467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:54:58.535508 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:54:58.537556 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:54:58.537581 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:54:58.537991 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:54:58.538307 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:54:58.540018 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:58.541496 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:58.543254 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:54:58.543979 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:54:58.544171 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:54:58.548466 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:54:58.548477 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:54:58.548493 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:54:58.548529 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:54:58.551966 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:58.552929 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:58.552936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:58.552942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:58.553113 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:54:58.553577 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:54:58.553582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:54:58.553585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:54:58.554263 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:54:58.554542 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 02:54:58.555127 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 02:54:58.555620 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":393,"threads_started":1}
I20260926 02:54:58.556957 31606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dcdbde380464052a1907d2c29fff760, State: Initialized, Role: FOLLOWER
I20260926 02:54:58.557668 31606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcdbde380464052a1907d2c29fff760" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37559 } }
I20260926 02:54:58.557999 31606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:54:58.558140 31606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:54:58.558320 31606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:54:58.560765 31606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcdbde380464052a1907d2c29fff760" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37559 } }
I20260926 02:54:58.561259 31606 leader_election.cc:302] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dcdbde380464052a1907d2c29fff760; no voters:
I20260926 02:54:58.561952 31606 leader_election.cc:288] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:54:58.562141 31611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:54:58.562984 31611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [term 1 LEADER]: Becoming Leader. State: Replica: 0dcdbde380464052a1907d2c29fff760, State: Running, Role: LEADER
I20260926 02:54:58.563604 31611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcdbde380464052a1907d2c29fff760" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37559 } }
I20260926 02:54:58.564163 31606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:54:58.566924 31612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dcdbde380464052a1907d2c29fff760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcdbde380464052a1907d2c29fff760" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37559 } } }
I20260926 02:54:58.567266 31612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [sys.catalog]: This master's current role is: LEADER
I20260926 02:54:58.569253 31618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:54:58.569105 31613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dcdbde380464052a1907d2c29fff760. Latest consensus state: current_term: 1 leader_uuid: "0dcdbde380464052a1907d2c29fff760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcdbde380464052a1907d2c29fff760" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 37559 } } }
I20260926 02:54:58.569442 31613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760 [sys.catalog]: This master's current role is: LEADER
I20260926 02:54:58.574999 31618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:54:58.575721 31606 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 02:54:58.595654 31618 catalog_manager.cc:1347] Generated new cluster ID: 4b24134a219249e791a79379d978eeea
I20260926 02:54:58.595847 31618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:54:58.606777 31618 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:54:58.608839 31618 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:54:58.621457 31618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760: Generated new TSK 0
I20260926 02:54:58.622637 31618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:54:59.466876 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.185.212:53435/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37559
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:54:59 dist-test-slave-ktng krb5kdc[31095](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790391299, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 02:54:59.904836 31651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:54:59.905171 31651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:54:59.905347 31651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:54:59.913000 31651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:54:59.913144 31651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:54:59.913334 31651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:54:59.924078 31651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.185.212:53435/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:54:59.925608 31651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:54:59.926946 31651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:54:59.945039 31661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:59.945746 31662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:54:59.952545 31664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:54:59.953562 31651 server_base.cc:1034] running on GCE node
I20260926 02:54:59.954380 31651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:54:59.959613 31651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:54:59.960929 31651 hybrid_clock.cc:648] HybridClock initialized: now 1790391299960839 us; error 92 us; skew 500 ppm
I20260926 02:54:59.961385 31651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:54:59 dist-test-slave-ktng krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391299, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:54:59.966488 31651 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:54:59.969676 31651 webserver.cc:460] Webserver started at http://127.19.185.193:34167/ using document root <none> and password file <none>
I20260926 02:54:59.970575 31651 fs_manager.cc:359] Metadata directory not provided
I20260926 02:54:59.970793 31651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:54:59.971138 31651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:54:59 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391299, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:54:59.991329 31651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "720eb552251748dcb7629e2939a78c55"
format_stamp: "Formatted at 2026-09-26 02:54:59 on dist-test-slave-ktng"
server_key: "4219920efb570137c084b49076b29692"
server_key_iv: "3ad162090e5ad4c0a1bce4c7b3ac3664"
server_key_version: "kuduclusterkey@0"
I20260926 02:54:59.992466 31651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "720eb552251748dcb7629e2939a78c55"
format_stamp: "Formatted at 2026-09-26 02:54:59 on dist-test-slave-ktng"
server_key: "4219920efb570137c084b49076b29692"
server_key_iv: "3ad162090e5ad4c0a1bce4c7b3ac3664"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:00.012053 31651 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.005s sys 0.012s
I20260926 02:55:00.030696 31671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:00.032842 31651 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 02:55:00.033116 31651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "720eb552251748dcb7629e2939a78c55"
format_stamp: "Formatted at 2026-09-26 02:54:59 on dist-test-slave-ktng"
server_key: "4219920efb570137c084b49076b29692"
server_key_iv: "3ad162090e5ad4c0a1bce4c7b3ac3664"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:00.033502 31651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:55:00.068785 31651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:55:00.073781 31651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:55:00.074100 31651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:55:00.075325 31651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:55:00.077083 31651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:55:00.077180 31651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:00.077296 31651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:55:00.077356 31651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:00.177438 31651 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:43901
I20260926 02:55:00.178033 31783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:43901 every 8 connection(s)
I20260926 02:55:00.179220 31651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:55:00.187974 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 31651
I20260926 02:55:00.188359 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:55:00.189277 20199 external_mini_cluster.cc:1428] Setting key 6833b824d17d2b1deaae9eba5c98bcb8
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
Sep 26 02:55:00 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391299, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
I20260926 02:55:00.237740 31787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:00.199482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:35897 (local address 127.19.185.254:37559)
0926 02:55:00.211633 (+ 12151us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:00.211640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:00.215207 (+ 3567us) server_negotiation.cc:407] Connection header received
0926 02:55:00.215286 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:00.215320 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:00.215418 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:00.215540 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:00.218033 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:00.219316 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:00.222092 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:00.222808 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:00.222928 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:00.230510 (+ 7582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:00.230517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:00.230522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:00.230571 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:00.234060 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:00.234987 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:00.234993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:00.234997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:00.235130 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:00.235670 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:00.235675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:00.235679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:00.236027 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:00.236139 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:00.237126 (+ 987us) server_negotiation.cc:299] Negotiation successful
0926 02:55:00.237354 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12022,"thread_start_us":12394,"threads_started":1}
I20260926 02:55:00.240592 31786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:00.191894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:37559 (local address 127.19.185.193:35897)
0926 02:55:00.211204 (+ 19310us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:00.211266 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:00.214754 (+ 3488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:00.216068 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:00.216082 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:00.216801 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:00.217760 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:00.217789 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:00.219556 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:00.219562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:00.221891 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:00.221917 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:00.223326 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:00.223335 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:00.223659 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:00.227522 (+ 3863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:00.227554 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:00.230217 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:00.234235 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:00.234245 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:00.234264 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:00.234846 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:00.235266 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:00.235270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:00.235274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:00.235543 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:00.236246 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:00.236254 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:00.236695 (+ 441us) client_negotiation.cc:769] Sending connection context
0926 02:55:00.238057 (+ 1362us) client_negotiation.cc:240] Negotiation successful
0926 02:55:00.238626 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18933,"thread_start_us":18704,"threads_started":1}
I20260926 02:55:00.242337 31784 heartbeater.cc:347] Connected to a master server at 127.19.185.254:37559
I20260926 02:55:00.242658 31784 heartbeater.cc:464] Registering TS with master...
I20260926 02:55:00.243537 31784 heartbeater.cc:511] Master 127.19.185.254:37559 requested a full tablet report, sending...
I20260926 02:55:00.245911 31571 ts_manager.cc:194] Registered new tserver with Master: 720eb552251748dcb7629e2939a78c55 (127.19.185.193:43901)
I20260926 02:55:00.247989 31571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:35897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:55:00.280149 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.185.212:53435/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37559
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:55:00 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391299, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
Sep 26 02:55:00 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391299, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
W20260926 02:55:00.766381 31790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:55:00.766870 31790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:55:00.767069 31790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:55:00.778339 31790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:55:00.778579 31790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:55:00.778777 31790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:55:00.794207 31790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.185.212:53435/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:55:00.796104 31790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:55:00.797780 31790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:55:00.817250 31797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:00.817325 31798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:00.832948 31800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:00.835608 31790 server_base.cc:1034] running on GCE node
I20260926 02:55:00.836542 31790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:55:00.837911 31790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:55:00.839341 31790 hybrid_clock.cc:648] HybridClock initialized: now 1790391300839188 us; error 92 us; skew 500 ppm
I20260926 02:55:00.839761 31790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:55:00 dist-test-slave-ktng krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391300, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:55:00.844411 31790 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:55:00.859828 31790 webserver.cc:460] Webserver started at http://127.19.185.194:46225/ using document root <none> and password file <none>
I20260926 02:55:00.860947 31790 fs_manager.cc:359] Metadata directory not provided
I20260926 02:55:00.861109 31790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:55:00.861699 31790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:55:00 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391300, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:55:00.880831 31790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "f0656487f2564df6b0627ec7a8d6c602"
format_stamp: "Formatted at 2026-09-26 02:55:00 on dist-test-slave-ktng"
server_key: "df61836dad572ef886cc8f190065b4d5"
server_key_iv: "c2f040a39f2ca9403350c62d2814629b"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:00.882174 31790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "f0656487f2564df6b0627ec7a8d6c602"
format_stamp: "Formatted at 2026-09-26 02:55:00 on dist-test-slave-ktng"
server_key: "df61836dad572ef886cc8f190065b4d5"
server_key_iv: "c2f040a39f2ca9403350c62d2814629b"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:00.888078 31790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:55:00.905366 31814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:00.907490 31790 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:55:00.907680 31790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "f0656487f2564df6b0627ec7a8d6c602"
format_stamp: "Formatted at 2026-09-26 02:55:00 on dist-test-slave-ktng"
server_key: "df61836dad572ef886cc8f190065b4d5"
server_key_iv: "c2f040a39f2ca9403350c62d2814629b"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:00.907948 31790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:55:00.934056 31790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:55:00.939522 31790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:55:00.939849 31790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:55:00.941087 31790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:55:00.942832 31790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:55:00.942929 31790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:00.943051 31790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:55:00.943115 31790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:01.002142 31790 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:42171
I20260926 02:55:01.002274 31926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:42171 every 8 connection(s)
I20260926 02:55:01.003559 31790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:55:01.009023 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 31790
I20260926 02:55:01.009253 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:55:01.009984 20199 external_mini_cluster.cc:1428] Setting key f54ba947877d04d2ace6a5332a4f9eff
Sep 26 02:55:01 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391300, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:55:01.028842 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:01.007645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:43285 (local address 127.19.185.254:37559)
0926 02:55:01.011898 (+ 4253us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:01.011905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:01.011923 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:55:01.011976 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:01.011981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:01.012069 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:01.012943 (+ 874us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:01.014100 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.015374 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.017885 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.018651 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.018800 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:01.022791 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:01.022796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:01.022800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:01.022831 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:01.025641 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:01.026460 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:01.026467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:01.026470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:01.026625 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:01.027047 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:01.027054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:01.027058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:01.027391 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:01.027548 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:01.028231 (+ 683us) server_negotiation.cc:299] Negotiation successful
0926 02:55:01.028447 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4131,"thread_start_us":4006,"threads_started":1}
I20260926 02:55:01.029881 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:01.007040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:37559 (local address 127.19.185.194:43285)
0926 02:55:01.007905 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:01.007963 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:01.011173 (+ 3210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:01.012463 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:01.012476 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:01.013080 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:01.013922 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:01.013949 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.015577 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.015583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:01.017701 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:01.017716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.018913 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.018921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:01.019198 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:01.020010 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:01.020038 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:01.022565 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:01.025883 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:01.025892 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:01.025912 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:01.026339 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:01.026738 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:01.026743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:01.026747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:01.026931 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:01.027518 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:01.027527 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:01.027864 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 02:55:01.028142 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 02:55:01.028674 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":383,"threads_started":1}
I20260926 02:55:01.031442 31927 heartbeater.cc:347] Connected to a master server at 127.19.185.254:37559
I20260926 02:55:01.031787 31927 heartbeater.cc:464] Registering TS with master...
I20260926 02:55:01.032512 31927 heartbeater.cc:511] Master 127.19.185.254:37559 requested a full tablet report, sending...
I20260926 02:55:01.034096 31571 ts_manager.cc:194] Registered new tserver with Master: f0656487f2564df6b0627ec7a8d6c602 (127.19.185.194:42171)
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
I20260926 02:55:01.035214 31571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:43285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:55:01.071414 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.185.212:53435/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37559
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:55:01.251664 31784 heartbeater.cc:503] Master 127.19.185.254:37559 was elected leader, sending a full tablet report...
W20260926 02:55:01.447883 31933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:55:01.448274 31933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:55:01.448460 31933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:55:01.456518 31933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:55:01.456631 31933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:55:01.456837 31933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:55:01.466967 31933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.185.212:53435/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:37559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:55:01.468276 31933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:55:01.469571 31933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:55:01.483469 31938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:01.483456 31941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:01.483682 31939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:01.484141 31933 server_base.cc:1034] running on GCE node
I20260926 02:55:01.484772 31933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:55:01.485707 31933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:55:01.486902 31933 hybrid_clock.cc:648] HybridClock initialized: now 1790391301486850 us; error 62 us; skew 500 ppm
I20260926 02:55:01.487360 31933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:55:01 dist-test-slave-ktng krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391301, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:55:01.491009 31933 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:55:01.492612 31933 webserver.cc:460] Webserver started at http://127.19.185.195:46815/ using document root <none> and password file <none>
I20260926 02:55:01.493319 31933 fs_manager.cc:359] Metadata directory not provided
I20260926 02:55:01.493450 31933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:55:01.493778 31933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:55:01 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391301, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:55:01.512004 31933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "7f0867037ea945b1b5737ee1a462a29c"
format_stamp: "Formatted at 2026-09-26 02:55:01 on dist-test-slave-ktng"
server_key: "48dc0742663865c91f8a374c1d1e660b"
server_key_iv: "6c9deaed0e159209f88c68f7d31bb5f3"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:01.513109 31933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "7f0867037ea945b1b5737ee1a462a29c"
format_stamp: "Formatted at 2026-09-26 02:55:01 on dist-test-slave-ktng"
server_key: "48dc0742663865c91f8a374c1d1e660b"
server_key_iv: "6c9deaed0e159209f88c68f7d31bb5f3"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:01.520393 31933 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 02:55:01.538112 31948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:01.540128 31933 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:55:01.540326 31933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "7f0867037ea945b1b5737ee1a462a29c"
format_stamp: "Formatted at 2026-09-26 02:55:01 on dist-test-slave-ktng"
server_key: "48dc0742663865c91f8a374c1d1e660b"
server_key_iv: "6c9deaed0e159209f88c68f7d31bb5f3"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:01.540606 31933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:55:01.574448 31933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:55:01.579602 31933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:55:01.579924 31933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:55:01.581067 31933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:55:01.582839 31933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:55:01.582926 31933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:01.583040 31933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:55:01.583097 31933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:01.618534 31933 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:45489
I20260926 02:55:01.618554 32060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:45489 every 8 connection(s)
I20260926 02:55:01.619935 31933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:55:01.626580 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 31933
I20260926 02:55:01.626761 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:55:01.627218 20199 external_mini_cluster.cc:1428] Setting key 62f62d684c124fe335a01d6637344c21
Sep 26 02:55:01 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391301, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:55:01.640655 32063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:01.622497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:43779 (local address 127.19.185.254:37559)
0926 02:55:01.622915 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:01.622920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:01.624084 (+ 1164us) server_negotiation.cc:407] Connection header received
0926 02:55:01.625872 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:01.625878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:01.625978 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:01.626134 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:01.627808 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.628701 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.630240 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.630694 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.630829 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:01.634081 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:01.634094 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:01.634099 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:01.634133 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:01.636974 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:01.637819 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:01.637825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:01.637836 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:01.637998 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:01.638630 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:01.638634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:01.638636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:01.638974 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:01.639132 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:01.640018 (+ 886us) server_negotiation.cc:299] Negotiation successful
0926 02:55:01.640267 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":218,"threads_started":1}
I20260926 02:55:01.641901 32064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:01.623116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:37559 (local address 127.19.185.195:43779)
0926 02:55:01.623903 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:01.623947 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:01.625582 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:01.626390 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:01.626399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:01.626951 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:01.627632 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:01.627651 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.628866 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.628871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:01.630089 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:01.630101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.630919 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.630926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:01.631164 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:01.631909 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:01.631936 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:01.633872 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:01.637150 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:01.637161 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:01.637180 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:01.637668 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:01.638125 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:01.638130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:01.638134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:01.638483 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:01.639121 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:01.639130 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:01.639577 (+ 447us) client_negotiation.cc:769] Sending connection context
0926 02:55:01.639865 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 02:55:01.640440 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":391,"threads_started":1}
I20260926 02:55:01.643422 32061 heartbeater.cc:347] Connected to a master server at 127.19.185.254:37559
I20260926 02:55:01.643738 32061 heartbeater.cc:464] Registering TS with master...
I20260926 02:55:01.644454 32061 heartbeater.cc:511] Master 127.19.185.254:37559 requested a full tablet report, sending...
I20260926 02:55:01.645860 31571 ts_manager.cc:194] Registered new tserver with Master: 7f0867037ea945b1b5737ee1a462a29c (127.19.185.195:45489)
I20260926 02:55:01.646677 31571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:43779
I20260926 02:55:01.654969 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:55:01.672677 32063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:01.659660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.19.185.254:37559)
0926 02:55:01.659816 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:01.659821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:01.660348 (+ 527us) server_negotiation.cc:407] Connection header received
0926 02:55:01.660557 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:01.660562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:01.660633 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:01.660739 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:01.662166 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.662900 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.664304 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.664924 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.665059 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:01.667473 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:01.667479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:01.667490 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:01.667523 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:01.669836 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:01.670611 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:01.670619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:01.670622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:01.670748 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:01.671140 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:01.671146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:01.671150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:01.671520 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:01.671775 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:01.672119 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 02:55:01.672348 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:55:01.676913 31571 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:55:01.684361 31571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:55:01.719995 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:01.696695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:45489 (local address 127.0.0.1:34508)
0926 02:55:01.708430 (+ 11735us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:01.708485 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:01.708698 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:01.711782 (+ 3084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:01.711790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:01.711840 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:55:01.712391 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:01.712412 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.714625 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.714631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:01.717137 (+ 2506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:01.717152 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.718895 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.718902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:01.719072 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:01.719124 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 02:55:01.719257 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 02:55:01.719477 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11466,"mutex_wait_us":132,"spinlock_wait_cycles":3968,"thread_start_us":13079,"threads_started":1}
I20260926 02:55:01.720661 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:01.710207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:42171 (local address 127.0.0.1:42822)
0926 02:55:01.712746 (+ 2539us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:01.712766 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:01.712882 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:01.713616 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:01.713620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:01.713641 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:55:01.713997 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:01.714006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.715962 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.715966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:01.718053 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:01.718063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.719828 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.719834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:01.720039 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:01.720089 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 02:55:01.720176 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 02:55:01.720290 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2361,"mutex_wait_us":28,"thread_start_us":2206,"threads_started":1}
I20260926 02:55:01.720747 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:01.696932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.19.185.195:45489)
0926 02:55:01.710820 (+ 13888us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:01.710829 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:01.710863 (+ 34us) server_negotiation.cc:407] Connection header received
0926 02:55:01.710971 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:01.710980 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:01.711272 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:01.711475 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:55:01.712658 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.714496 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.717285 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.718689 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.718881 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:01.719068 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:01.720224 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0926 02:55:01.720404 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13731,"thread_start_us":13577,"threads_started":1}
I20260926 02:55:01.721005 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:01.697326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.19.185.193:43901)
0926 02:55:01.709350 (+ 12024us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:01.709363 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:01.710301 (+ 938us) server_negotiation.cc:407] Connection header received
0926 02:55:01.711029 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:01.711037 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:01.711266 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:01.711472 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:55:01.712994 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.714540 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.717720 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.719152 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.719367 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:01.719567 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:01.720406 (+ 839us) server_negotiation.cc:299] Negotiation successful
0926 02:55:01.720617 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11770,"thread_start_us":12031,"threads_started":1}
I20260926 02:55:01.721309 32079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:01.710452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.19.185.194:42171)
0926 02:55:01.712240 (+ 1788us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:01.712249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:01.713000 (+ 751us) server_negotiation.cc:407] Connection header received
0926 02:55:01.713107 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:01.713113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:01.713424 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:01.713612 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:55:01.714162 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.715795 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.718194 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.719369 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.719497 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:01.719655 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:01.720884 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0926 02:55:01.721049 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1626,"thread_start_us":1517,"threads_started":1}
I20260926 02:55:01.721830 32074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:01.697083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:43901 (local address 127.0.0.1:44586)
0926 02:55:01.710144 (+ 13061us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:01.710169 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:01.710824 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:01.711782 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:01.711790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:01.711835 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:55:01.712780 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:01.712795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.714703 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.714708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:01.717551 (+ 2843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:01.717566 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:01.719724 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:01.719730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:01.719926 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:01.719959 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:55:01.721293 (+ 1334us) client_negotiation.cc:240] Negotiation successful
0926 02:55:01.721424 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12888,"mutex_wait_us":851,"spinlock_wait_cycles":896,"thread_start_us":10903,"threads_started":1}
I20260926 02:55:01.724489 31996 tablet_service.cc:1467] Processing CreateTablet for tablet 261045f797f64a6b8f214598c44d4dff (DEFAULT_TABLE table=test-table [id=9ac599d220db46d9a8888a25851ae968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:55:01.724900 31862 tablet_service.cc:1467] Processing CreateTablet for tablet 261045f797f64a6b8f214598c44d4dff (DEFAULT_TABLE table=test-table [id=9ac599d220db46d9a8888a25851ae968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:55:01.725755 31996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 261045f797f64a6b8f214598c44d4dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:55:01.725323 31719 tablet_service.cc:1467] Processing CreateTablet for tablet 261045f797f64a6b8f214598c44d4dff (DEFAULT_TABLE table=test-table [id=9ac599d220db46d9a8888a25851ae968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:55:01.726254 31862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 261045f797f64a6b8f214598c44d4dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:55:01.726686 31719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 261045f797f64a6b8f214598c44d4dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:55:01.737716 32080 tablet_bootstrap.cc:492] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c: Bootstrap starting.
I20260926 02:55:01.741231 32080 tablet_bootstrap.cc:654] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c: Neither blocks nor log segments found. Creating new log.
I20260926 02:55:01.741274 32081 tablet_bootstrap.cc:492] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602: Bootstrap starting.
I20260926 02:55:01.742636 32080 log.cc:824] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:55:01.745478 32081 tablet_bootstrap.cc:654] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602: Neither blocks nor log segments found. Creating new log.
I20260926 02:55:01.747008 32081 log.cc:824] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602: Log is configured to *not* fsync() on all Append() calls
I20260926 02:55:01.748351 32082 tablet_bootstrap.cc:492] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55: Bootstrap starting.
I20260926 02:55:01.750494 32081 tablet_bootstrap.cc:492] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602: No bootstrap required, opened a new log
I20260926 02:55:01.750813 32081 ts_tablet_manager.cc:1397] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260926 02:55:01.752027 32082 tablet_bootstrap.cc:654] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55: Neither blocks nor log segments found. Creating new log.
I20260926 02:55:01.752586 32080 tablet_bootstrap.cc:492] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c: No bootstrap required, opened a new log
I20260926 02:55:01.752902 32080 ts_tablet_manager.cc:1397] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.005s
I20260926 02:55:01.753700 32082 log.cc:824] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55: Log is configured to *not* fsync() on all Append() calls
I20260926 02:55:01.756824 32082 tablet_bootstrap.cc:492] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55: No bootstrap required, opened a new log
I20260926 02:55:01.757300 32082 ts_tablet_manager.cc:1397] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.005s
I20260926 02:55:01.766800 32081 raft_consensus.cc:348] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0867037ea945b1b5737ee1a462a29c" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45489 } } peers { permanent_uuid: "720eb552251748dcb7629e2939a78c55" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43901 } } peers { permanent_uuid: "f0656487f2564df6b0627ec7a8d6c602" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42171 } }
I20260926 02:55:01.767681 32081 raft_consensus.cc:374] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:55:01.767848 32081 raft_consensus.cc:729] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0656487f2564df6b0627ec7a8d6c602, State: Initialized, Role: FOLLOWER
I20260926 02:55:01.768748 32081 consensus_queue.cc:260] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0867037ea945b1b5737ee1a462a29c" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45489 } } peers { permanent_uuid: "720eb552251748dcb7629e2939a78c55" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43901 } } peers { permanent_uuid: "f0656487f2564df6b0627ec7a8d6c602" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42171 } }
I20260926 02:55:01.769768 32080 raft_consensus.cc:348] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0867037ea945b1b5737ee1a462a29c" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45489 } } peers { permanent_uuid: "720eb552251748dcb7629e2939a78c55" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43901 } } peers { permanent_uuid: "f0656487f2564df6b0627ec7a8d6c602" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42171 } }
I20260926 02:55:01.769898 32082 raft_consensus.cc:348] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0867037ea945b1b5737ee1a462a29c" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45489 } } peers { permanent_uuid: "720eb552251748dcb7629e2939a78c55" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43901 } } peers { permanent_uuid: "f0656487f2564df6b0627ec7a8d6c602" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42171 } }
I20260926 02:55:01.770624 32080 raft_consensus.cc:374] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:55:01.770829 32080 raft_consensus.cc:729] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f0867037ea945b1b5737ee1a462a29c, State: Initialized, Role: FOLLOWER
I20260926 02:55:01.771380 31927 heartbeater.cc:503] Master 127.19.185.254:37559 was elected leader, sending a full tablet report...
I20260926 02:55:01.771390 32082 raft_consensus.cc:374] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:55:01.771637 32082 raft_consensus.cc:729] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720eb552251748dcb7629e2939a78c55, State: Initialized, Role: FOLLOWER
I20260926 02:55:01.772338 32081 ts_tablet_manager.cc:1428] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602: Time spent starting tablet: real 0.021s user 0.020s sys 0.000s
I20260926 02:55:01.772341 32082 consensus_queue.cc:260] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0867037ea945b1b5737ee1a462a29c" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45489 } } peers { permanent_uuid: "720eb552251748dcb7629e2939a78c55" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43901 } } peers { permanent_uuid: "f0656487f2564df6b0627ec7a8d6c602" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42171 } }
I20260926 02:55:01.774487 32082 ts_tablet_manager.cc:1428] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55: Time spent starting tablet: real 0.017s user 0.010s sys 0.005s
I20260926 02:55:01.773653 32080 consensus_queue.cc:260] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0867037ea945b1b5737ee1a462a29c" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45489 } } peers { permanent_uuid: "720eb552251748dcb7629e2939a78c55" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43901 } } peers { permanent_uuid: "f0656487f2564df6b0627ec7a8d6c602" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42171 } }
I20260926 02:55:01.776798 32061 heartbeater.cc:503] Master 127.19.185.254:37559 was elected leader, sending a full tablet report...
I20260926 02:55:01.779152 32080 ts_tablet_manager.cc:1428] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c: Time spent starting tablet: real 0.026s user 0.011s sys 0.011s
W20260926 02:55:01.871611 32062 tablet.cc:2369] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:55:01.940240 31785 tablet.cc:2369] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:55:02.007447 31928 tablet.cc:2369] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:55:02.034204 32088 raft_consensus.cc:482] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:55:02.034513 32088 raft_consensus.cc:504] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0867037ea945b1b5737ee1a462a29c" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45489 } } peers { permanent_uuid: "720eb552251748dcb7629e2939a78c55" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43901 } } peers { permanent_uuid: "f0656487f2564df6b0627ec7a8d6c602" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42171 } }
I20260926 02:55:02.036461 32088 leader_election.cc:288] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720eb552251748dcb7629e2939a78c55 (127.19.185.193:43901), f0656487f2564df6b0627ec7a8d6c602 (127.19.185.194:42171)
I20260926 02:55:02.044741 32064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:02.037379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:43901 (local address 127.19.185.195:52639)
0926 02:55:02.037543 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:02.037561 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:02.037704 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:02.038197 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:02.038205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:02.038241 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:55:02.038573 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:02.038588 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.039937 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.039942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:02.042111 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:02.042123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.043849 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.043856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:02.044163 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:02.044212 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 02:55:02.044301 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 02:55:02.044447 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":19}
I20260926 02:55:02.045243 32079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:02.037403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:38683 (local address 127.19.185.194:42171)
0926 02:55:02.037719 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:02.037728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:02.038161 (+ 433us) server_negotiation.cc:407] Connection header received
0926 02:55:02.038389 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:02.038394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:02.038519 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:02.038636 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:55:02.039342 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.040389 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.043010 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.044137 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.044294 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:02.044366 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:02.044791 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 02:55:02.044928 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:55:02.045353 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:02.036991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:52639 (local address 127.19.185.193:43901)
0926 02:55:02.037237 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:02.037245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:02.037633 (+ 388us) server_negotiation.cc:407] Connection header received
0926 02:55:02.037919 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:02.037924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:02.038003 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:02.038109 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:55:02.038732 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.039760 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.042287 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.043681 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.043821 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:02.043874 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:02.044367 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 02:55:02.044515 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:55:02.045575 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:02.037441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:42171 (local address 127.19.185.195:38683)
0926 02:55:02.038036 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:02.038054 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:02.038165 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:02.038700 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:02.038706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:02.038731 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:55:02.039120 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:02.039132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.040571 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.040575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:02.042853 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:02.042867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.044404 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.044410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:02.044605 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:02.044640 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:55:02.044740 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 02:55:02.044903 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":1,"thread_start_us":306,"threads_started":1}
I20260926 02:55:02.046861 31739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261045f797f64a6b8f214598c44d4dff" candidate_uuid: "7f0867037ea945b1b5737ee1a462a29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720eb552251748dcb7629e2939a78c55" is_pre_election: true
I20260926 02:55:02.046931 31882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261045f797f64a6b8f214598c44d4dff" candidate_uuid: "7f0867037ea945b1b5737ee1a462a29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0656487f2564df6b0627ec7a8d6c602" is_pre_election: true
I20260926 02:55:02.047747 31739 raft_consensus.cc:2454] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f0867037ea945b1b5737ee1a462a29c in term 0.
I20260926 02:55:02.047748 31882 raft_consensus.cc:2454] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f0867037ea945b1b5737ee1a462a29c in term 0.
I20260926 02:55:02.048581 31952 leader_election.cc:302] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f0867037ea945b1b5737ee1a462a29c, f0656487f2564df6b0627ec7a8d6c602; no voters:
I20260926 02:55:02.049120 32088 raft_consensus.cc:2781] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:55:02.049345 32088 raft_consensus.cc:482] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:55:02.049503 32088 raft_consensus.cc:3037] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:55:02.053831 32088 raft_consensus.cc:504] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0867037ea945b1b5737ee1a462a29c" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45489 } } peers { permanent_uuid: "720eb552251748dcb7629e2939a78c55" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43901 } } peers { permanent_uuid: "f0656487f2564df6b0627ec7a8d6c602" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42171 } }
I20260926 02:55:02.055013 32088 leader_election.cc:288] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [CANDIDATE]: Term 1 election: Requested vote from peers 720eb552251748dcb7629e2939a78c55 (127.19.185.193:43901), f0656487f2564df6b0627ec7a8d6c602 (127.19.185.194:42171)
I20260926 02:55:02.055577 31739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261045f797f64a6b8f214598c44d4dff" candidate_uuid: "7f0867037ea945b1b5737ee1a462a29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720eb552251748dcb7629e2939a78c55"
I20260926 02:55:02.055728 31882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "261045f797f64a6b8f214598c44d4dff" candidate_uuid: "7f0867037ea945b1b5737ee1a462a29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0656487f2564df6b0627ec7a8d6c602"
I20260926 02:55:02.055907 31739 raft_consensus.cc:3037] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:55:02.056017 31882 raft_consensus.cc:3037] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:55:02.060730 31882 raft_consensus.cc:2454] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f0867037ea945b1b5737ee1a462a29c in term 1.
I20260926 02:55:02.060748 31739 raft_consensus.cc:2454] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f0867037ea945b1b5737ee1a462a29c in term 1.
I20260926 02:55:02.061270 31952 leader_election.cc:302] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 720eb552251748dcb7629e2939a78c55, 7f0867037ea945b1b5737ee1a462a29c, f0656487f2564df6b0627ec7a8d6c602; no voters:
I20260926 02:55:02.061717 32088 raft_consensus.cc:2781] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:55:02.062513 32088 raft_consensus.cc:686] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [term 1 LEADER]: Becoming Leader. State: Replica: 7f0867037ea945b1b5737ee1a462a29c, State: Running, Role: LEADER
I20260926 02:55:02.063131 32088 consensus_queue.cc:237] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0867037ea945b1b5737ee1a462a29c" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45489 } } peers { permanent_uuid: "720eb552251748dcb7629e2939a78c55" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43901 } } peers { permanent_uuid: "f0656487f2564df6b0627ec7a8d6c602" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42171 } }
I20260926 02:55:02.068545 31569 catalog_manager.cc:5495] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f0867037ea945b1b5737ee1a462a29c (127.19.185.195). New cstate: current_term: 1 leader_uuid: "7f0867037ea945b1b5737ee1a462a29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0867037ea945b1b5737ee1a462a29c" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "720eb552251748dcb7629e2939a78c55" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0656487f2564df6b0627ec7a8d6c602" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42171 } health_report { overall_health: UNKNOWN } } }
I20260926 02:55:02.099792 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:02.093458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.19.185.195:45489)
0926 02:55:02.093701 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:02.093705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:02.093844 (+ 139us) server_negotiation.cc:407] Connection header received
0926 02:55:02.094029 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:02.094033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:02.094118 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:02.094245 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:55:02.094638 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.095622 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.097424 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.097965 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.098059 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:02.098736 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:55:02.099219 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:55:02.099425 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:02.099466 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 02:55:02.099527 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:55:02.109790 31882 raft_consensus.cc:1261] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602 [term 1 FOLLOWER]: Refusing update from remote peer 7f0867037ea945b1b5737ee1a462a29c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:55:02.109809 31739 raft_consensus.cc:1261] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55 [term 1 FOLLOWER]: Refusing update from remote peer 7f0867037ea945b1b5737ee1a462a29c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:55:02.110891 32088 consensus_queue.cc:1035] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0656487f2564df6b0627ec7a8d6c602" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:55:02.111286 32090 consensus_queue.cc:1035] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "720eb552251748dcb7629e2939a78c55" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 43901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:55:02.147238 31569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:55:02.147712 31569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:55:02.151698 31569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0dcdbde380464052a1907d2c29fff760: Sending DeleteTablet for 3 replicas of tablet 261045f797f64a6b8f214598c44d4dff
I20260926 02:55:02.152922 31862 tablet_service.cc:1514] Processing DeleteTablet for tablet 261045f797f64a6b8f214598c44d4dff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:55:02 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:42822
I20260926 02:55:02.152921 31719 tablet_service.cc:1514] Processing DeleteTablet for tablet 261045f797f64a6b8f214598c44d4dff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:55:02 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:44586
I20260926 02:55:02.152904 31996 tablet_service.cc:1514] Processing DeleteTablet for tablet 261045f797f64a6b8f214598c44d4dff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:55:02 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:34508
I20260926 02:55:02.154199 32105 tablet_replica.cc:330] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602: stopping tablet replica
I20260926 02:55:02.154668 32105 raft_consensus.cc:2229] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:55:02.155236 32105 raft_consensus.cc:2258] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:55:02.155954 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 31651
I20260926 02:55:02.157500 32105 ts_tablet_manager.cc:1905] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:55:02.164983 32105 ts_tablet_manager.cc:1918] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:55:02.165221 32105 log.cc:1196] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/261045f797f64a6b8f214598c44d4dff
I20260926 02:55:02.165518 32104 tablet_replica.cc:330] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c: stopping tablet replica
I20260926 02:55:02.166018 32105 ts_tablet_manager.cc:1939] T 261045f797f64a6b8f214598c44d4dff P f0656487f2564df6b0627ec7a8d6c602: Deleting consensus metadata
I20260926 02:55:02.166204 32104 raft_consensus.cc:2229] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:55:02.167079 32104 raft_consensus.cc:2258] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:55:02.168088 31558 catalog_manager.cc:4841] TS f0656487f2564df6b0627ec7a8d6c602 (127.19.185.194:42171): tablet 261045f797f64a6b8f214598c44d4dff (table test-table [id=9ac599d220db46d9a8888a25851ae968]) successfully deleted
I20260926 02:55:02.172003 32104 ts_tablet_manager.cc:1905] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:55:02.172071 32103 tablet_replica.cc:330] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55: stopping tablet replica
I20260926 02:55:02.172760 32103 raft_consensus.cc:2229] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:55:02.173517 32103 raft_consensus.cc:2258] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:55:02.179354 32104 ts_tablet_manager.cc:1918] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:55:02.179517 32104 log.cc:1196] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/261045f797f64a6b8f214598c44d4dff
I20260926 02:55:02.180027 32104 ts_tablet_manager.cc:1939] T 261045f797f64a6b8f214598c44d4dff P 7f0867037ea945b1b5737ee1a462a29c: Deleting consensus metadata
I20260926 02:55:02.181591 31558 catalog_manager.cc:4841] TS 7f0867037ea945b1b5737ee1a462a29c (127.19.185.195:45489): tablet 261045f797f64a6b8f214598c44d4dff (table test-table [id=9ac599d220db46d9a8888a25851ae968]) successfully deleted
I20260926 02:55:02.183161 32103 ts_tablet_manager.cc:1905] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 02:55:02 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:55:02.190506 32103 ts_tablet_manager.cc:1918] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:55:02.190672 32103 log.cc:1196] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/261045f797f64a6b8f214598c44d4dff
I20260926 02:55:02.191391 32103 ts_tablet_manager.cc:1939] T 261045f797f64a6b8f214598c44d4dff P 720eb552251748dcb7629e2939a78c55: Deleting consensus metadata
I20260926 02:55:02.193320 31557 catalog_manager.cc:4841] TS 720eb552251748dcb7629e2939a78c55 (127.19.185.193:43901): tablet 261045f797f64a6b8f214598c44d4dff (table test-table [id=9ac599d220db46d9a8888a25851ae968]) successfully deleted
I20260926 02:55:02.197032 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:02.157893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.19.185.193:43901)
0926 02:55:02.158133 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:02.158142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:02.171473 (+ 13331us) server_negotiation.cc:407] Connection header received
0926 02:55:02.172027 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:02.172033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:02.172128 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:02.172212 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:02.174079 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.175248 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.182018 (+ 6770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.182759 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.182875 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:02.188613 (+ 5738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:02.188627 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:02.188646 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:02.188690 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:02.192944 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:02.194750 (+ 1806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:02.194759 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:02.194763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:02.194916 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:02.195369 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:02.195374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:02.195377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:02.195814 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:02.196273 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:02.196431 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 02:55:02.196643 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 02:55:02.197685 31779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:55:02.289467 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 31651
I20260926 02:55:02.311771 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 31790
Sep 26 02:55:02 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:55:02.327672 32079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:02.312614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.19.185.194:42171)
0926 02:55:02.312799 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:02.312807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:02.312828 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:55:02.313079 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:02.313084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:02.313204 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:02.313313 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:02.314693 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.315537 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.316806 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.317203 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.317338 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:02.321660 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:02.321676 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:02.321693 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:02.321727 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:02.324828 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:02.325576 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:02.325584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:02.325586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:02.325713 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:02.326096 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:02.326102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:02.326105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:02.326459 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:02.326778 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:02.327065 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 02:55:02.327282 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:55:02.328411 31922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:55:02.418273 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 31790
I20260926 02:55:02.435662 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 31933
Sep 26 02:55:02 dist-test-slave-ktng krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:55:02.454645 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:02.436601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.19.185.195:45489)
0926 02:55:02.436816 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:02.436823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:02.436846 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:55:02.436921 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:02.436926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:02.437017 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:02.437135 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:02.438531 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.439908 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.441368 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:02.442078 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:02.442230 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:02.447081 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:02.447104 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:02.447128 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:02.447183 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:02.451634 (+ 4451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:02.452403 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:02.452413 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:02.452417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:02.452634 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:02.453003 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:02.453009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:02.453012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:02.453462 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:02.453794 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:02.454000 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 02:55:02.454230 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:55:02.455559 32056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:55:02.578260 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 31933
I20260926 02:55:02.593928 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 31539
I20260926 02:55:02.595235 31601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:55:02.707240 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 31539
I20260926 02:55:02.735702 20199 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 02:55:03.081257 20199 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 02:55:03.081524 20199 mini_ranger.cc:66] Stopping Ranger...
I20260926 02:55:03.427244 20199 mini_ranger.cc:68] Stopped Ranger
2026-09-26 02:55:03.427 UTC [31125] LOG: received smart shutdown request
2026-09-26 02:55:03.432 UTC [31125] LOG: background worker "logical replication launcher" (PID 31134) exited with exit code 1
2026-09-26 02:55:03.432 UTC [31129] LOG: shutting down
2026-09-26 02:55:03.623 UTC [31125] LOG: database system is shut down
2026-09-26T02:55:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51957 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:55:03 dist-test-slave-ktng krb5kdc[32119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:55:03 dist-test-slave-ktng krb5kdc[32119](info): set up 2 sockets
Sep 26 02:55:03 dist-test-slave-ktng krb5kdc[32119](info): commencing operation
krb5kdc: starting...
W20260926 02:55:04.968551 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:55:05 dist-test-slave-ktng krb5kdc[32119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:55:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:55:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.185.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.185.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/rangeradmin_127.19.185.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/rangeradmin_127.19.185.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.185.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.185.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/rangerlookup_127.19.185.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/rangerlookup_127.19.185.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.185.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.185.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/HTTP_127.19.185.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/HTTP_127.19.185.212@KRBTEST.COM.keytab.
I20260926 02:55:05.300463 20199 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAXkdvX/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/postgres -l logfile start
2026-09-26 02:55:08.041 UTC [32149] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 02:55:08.041 UTC [32149] LOG: listening on IPv4 address "127.19.185.212", port 57627
2026-09-26 02:55:08.045 UTC [32149] LOG: listening on Unix socket "/tmp/.s.PGSQL.57627"
2026-09-26 02:55:08.067 UTC [32152] LOG: database system was shut down at 2026-09-26 02:55:06 UTC
2026-09-26 02:55:08.071 UTC [32149] LOG: database system is ready to accept connections
I20260926 02:55:09.060317 20199 mini_postgres.cc:96] Postgres bound to 57627
2026-09-26 02:55:09.078 UTC [32160] FATAL: database "slave" does not exist
127.19.185.212:57627 - accepting connections
I20260926 02:55:09.078990 20199 mini_ranger.cc:161] Starting Ranger...
I20260926 02:55:09.106693 20199 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 02:55:09.634459 20199 mini_ranger.cc:87] Created ranger Postgres database
I20260926 02:55:09.634554 20199 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-admin
2026-09-26 02:55:09,918 [I] DB FLAVOR :POSTGRES
2026-09-26 02:55:09,919 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:55:09,919 [I] Checking connection..
2026-09-26 02:55:09,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 02:55:10,183 [I] Checking connection passed.
2026-09-26 02:55:10,184 [I] --------- Verifying version history table ---------
2026-09-26 02:55:10,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:55:10,446 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:55:10,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:55:10,690 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:55:10,690 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 02:55:10,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 02:55:10.945 UTC [32224] WARNING: there is no transaction in progress
2026-09-26 02:55:10,953 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 02:55:10,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:55:11,211 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 02:55:11,212 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:55:11,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 02:55:11,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 02:55:11,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ktng.c.gcp-upstream.internal','N') ;"
2026-09-26 02:55:11,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:55:12,198 [I] Table x_portal_user does not exist in database ranger
2026-09-26 02:55:12,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:55:12,444 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 02:55:12,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:55:12,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:55:12,947 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 02:55:12,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 02:55:13.937 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:13.955 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:13.972 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:13.989 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:14.006 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:14.024 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:14.080 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:14.087 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:14.102 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:14.113 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:14.123 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:14.139 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:14.149 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:14.156 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:14.163 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:14.170 UTC [32332] WARNING: there is no transaction in progress
2026-09-26 02:55:14,855 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 02:55:14,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 02:55:15,115 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 02:55:15,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:55:15,362 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 02:55:15,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:55:15,627 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 02:55:15,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:55:15,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:55:16,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ktng.c.gcp-upstream.internal';"
2026-09-26 02:55:16,382 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 02:55:16,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 02:55:16,641 [I] DB_PATCHES have already been applied
I20260926 02:55:16.647192 20199 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAXkdvX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 02:55:16.647275 20199 mini_ranger.cc:193] Using host: 127.19.185.212
I20260926 02:55:16.661820 20199 mini_ranger.cc:227] Ranger admin URL: http://127.19.185.212:53223
Sep 26, 2026 2:55:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:55:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:55:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:55:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:55:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:55:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:55:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 02:55:17.838564 20199 mini_ranger.cc:160] Time spent starting Ranger: real 8.760s user 0.010s sys 0.029s
Sep 26, 2026 2:55:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:55:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 2:55:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:55:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:55:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:55:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:55:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:55:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:55:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:55:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:55:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:55:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:55:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:55:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:55:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:55:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:55:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:55:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 02:55:38.731966 20199 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.185.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.185.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/rangerkms_127.19.185.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.185.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/rangerkms_127.19.185.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.185.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.185.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.185.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/HTTP_127.19.185.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.185.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/HTTP_127.19.185.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 02:55:38 dist-test-slave-ktng krb5kdc[32119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391338, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 02:55:38.924219 20199 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 02:55:38.948186 20199 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 02:55:39.932914 20199 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 02:55:39.933050 20199 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-kms
I20260926 02:55:39.933101 20199 mini_ranger_kms.cc:227] Using postgres at 127.19.185.212:57627
2026-09-26 02:55:40,293 [I] DB FLAVOR :POSTGRES
2026-09-26 02:55:40,294 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:55:40,294 [I] Checking connection
2026-09-26 02:55:40,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 02:55:40,558 [I] connection success
2026-09-26 02:55:40,559 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 02:55:40,559 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 02:55:40,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 02:55:40,793 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 02:55:40,793 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:55:40,793 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 02:55:40,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.185.212:57627/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 02:55:41,060 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 02:55:41.502564 20199 mini_ranger_kms.cc:310] Created kms service
I20260926 02:55:41.694414 20199 mini_ranger_kms.cc:326] Created kudu user
I20260926 02:55:41.739862 20199 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 02:55:41 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391338, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:55:42.139559 20199 mini_ranger_kms.cc:384] Added ranger policy
I20260926 02:55:42.139885 20199 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAXkdvX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAXkdvX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 02:55:42.139932 20199 mini_ranger_kms.cc:240] Using host: 127.19.185.212
I20260926 02:55:42.161345 20199 mini_ranger_kms.cc:276] Ranger KMS PID: 32512
I20260926 02:55:42.161477 20199 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.185.212:52161
Sep 26, 2026 2:55:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:55:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:55:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:55:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:55:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:55:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 02:55:44 dist-test-slave-ktng krb5kdc[32119](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790391344, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.185.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:55:44.486776 20199 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.563s user 0.068s sys 0.314s
I20260926 02:55:44.487022 20199 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 02:55:44.487224 20199 mini_ranger_kms.cc:401] 127.19.185.212:52161/kms/v1/keys
Sep 26 02:55:44 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391344, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.185.212@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
Sep 26 02:55:45 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391344, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.185.212@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
Sep 26 02:55:47 dist-test-slave-ktng krb5kdc[32119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:55:47.599447 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:41781
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:41781
--ranger_config_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.185.212:52161/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:55:47.938553 32563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:55:47.938899 32563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:55:47.939054 32563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:55:47.946374 32563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:55:47.946520 32563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:55:47.946626 32563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:55:47.946727 32563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:55:47.946817 32563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:55:47.957317 32563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.185.212:52161/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:41781
--ranger_config_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:41781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:55:47.958662 32563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:55:47.960255 32563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:55:47.974642 32568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:47.974607 32569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:47.975169 32571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:47.975455 32563 server_base.cc:1034] running on GCE node
I20260926 02:55:47.976428 32563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:55:47.978308 32563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:55:47.979568 32563 hybrid_clock.cc:648] HybridClock initialized: now 1790391347979513 us; error 67 us; skew 500 ppm
I20260926 02:55:47.979988 32563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:55:47 dist-test-slave-ktng krb5kdc[32119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391347, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:55:47.988430 32563 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:55:47.990130 32563 webserver.cc:460] Webserver started at http://127.19.185.254:37309/ using document root <none> and password file <none>
I20260926 02:55:47.990826 32563 fs_manager.cc:359] Metadata directory not provided
I20260926 02:55:47.990954 32563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:55:47.991353 32563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:55:47 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391347, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:55:48.041714 32563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc"
format_stamp: "Formatted at 2026-09-26 02:55:48 on dist-test-slave-ktng"
server_key: "3cefbdf53e711a41161ed28e131edcd3"
server_key_iv: "b3a2cb41ab80e28b5e61d2645565a781"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:48.042593 32563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc"
format_stamp: "Formatted at 2026-09-26 02:55:48 on dist-test-slave-ktng"
server_key: "3cefbdf53e711a41161ed28e131edcd3"
server_key_iv: "b3a2cb41ab80e28b5e61d2645565a781"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:48.048228 32563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:55:48.064831 32578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:48.066390 32563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:55:48.066573 32563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc"
format_stamp: "Formatted at 2026-09-26 02:55:48 on dist-test-slave-ktng"
server_key: "3cefbdf53e711a41161ed28e131edcd3"
server_key_iv: "b3a2cb41ab80e28b5e61d2645565a781"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:48.066802 32563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:55:48.094888 32563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:55:48.099021 32563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:55:48.099370 32563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:55:48.120651 32563 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:41781
I20260926 02:55:48.120687 32629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:41781 every 8 connection(s)
I20260926 02:55:48.122157 32563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:55:48.126159 32630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:55:48.127490 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 32563
I20260926 02:55:48.127702 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:55:48.128566 20199 external_mini_cluster.cc:1428] Setting key 16c597df145b306b3c34f8a43934f6f9
I20260926 02:55:48.144239 32630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc: Bootstrap starting.
I20260926 02:55:48.148643 32630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc: Neither blocks nor log segments found. Creating new log.
I20260926 02:55:48.149955 32630 log.cc:824] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc: Log is configured to *not* fsync() on all Append() calls
I20260926 02:55:48.153030 32630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc: No bootstrap required, opened a new log
Sep 26 02:55:48 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:55:48.162984 32630 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41781 } }
I20260926 02:55:48.163453 32630 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:55:48.163589 32630 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7eda3c2b2ae44b0bb5ccf46f4c740fc, State: Initialized, Role: FOLLOWER
I20260926 02:55:48.164280 32630 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41781 } }
I20260926 02:55:48.164537 32630 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:55:48.164671 32630 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:55:48.164855 32630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:55:48.168102 32630 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41781 } }
I20260926 02:55:48.168686 32630 leader_election.cc:302] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7eda3c2b2ae44b0bb5ccf46f4c740fc; no voters:
I20260926 02:55:48.169404 32630 leader_election.cc:288] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:55:48.169466 32633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:48.130856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.19.185.254:41781)
0926 02:55:48.143963 (+ 13107us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:48.143983 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:48.144483 (+ 500us) server_negotiation.cc:407] Connection header received
0926 02:55:48.147086 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:48.147116 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:48.147658 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:48.148112 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:48.150216 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:48.152287 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:48.154188 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:48.155220 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:48.155443 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:48.160527 (+ 5084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:48.160537 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:48.160561 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:48.160606 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:48.164540 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:48.165383 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:48.165393 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:48.165402 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:48.165585 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:48.166052 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:48.166059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:48.166063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:48.166858 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:48.167177 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:48.167619 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 02:55:48.168046 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12813,"thread_start_us":12546,"threads_started":1}
I20260926 02:55:48.169561 32635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:55:48.176443 32635 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 1 LEADER]: Becoming Leader. State: Replica: b7eda3c2b2ae44b0bb5ccf46f4c740fc, State: Running, Role: LEADER
I20260926 02:55:48.177115 32635 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41781 } }
I20260926 02:55:48.177486 32630 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:55:48.180397 32637 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7eda3c2b2ae44b0bb5ccf46f4c740fc. Latest consensus state: current_term: 1 leader_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41781 } } }
I20260926 02:55:48.180778 32637 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [sys.catalog]: This master's current role is: LEADER
I20260926 02:55:48.180482 32636 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41781 } } }
I20260926 02:55:48.180904 32636 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [sys.catalog]: This master's current role is: LEADER
I20260926 02:55:48.181649 32641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:55:48.195349 32641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:55:48.222506 32641 catalog_manager.cc:1347] Generated new cluster ID: ccc3f88e210740f1b9ca8b87ae0db489
I20260926 02:55:48.222631 32641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:55:48.250594 32641 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:55:48.252771 32641 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:55:48.255646 32630 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 02:55:48.281112 32641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc: Generated new TSK 0
I20260926 02:55:48.282763 32641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:55:49.099577 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.185.212:52161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:41781
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:55:49 dist-test-slave-ktng krb5kdc[32119](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790391349, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 02:55:49.535778 32675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:55:49.536206 32675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:55:49.536378 32675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:55:49.548168 32675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:55:49.548316 32675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:55:49.548586 32675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:55:49.564374 32675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.185.212:52161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:41781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:55:49.566280 32675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:55:49.568114 32675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:55:49.585122 32685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:49.586876 32686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:49.605127 32688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:49.605964 32675 server_base.cc:1034] running on GCE node
I20260926 02:55:49.606786 32675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:55:49.608120 32675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:55:49.609416 32675 hybrid_clock.cc:648] HybridClock initialized: now 1790391349609339 us; error 62 us; skew 500 ppm
I20260926 02:55:49.609814 32675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:55:49 dist-test-slave-ktng krb5kdc[32119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391349, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:55:49.614084 32675 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:55:49.616865 32675 webserver.cc:460] Webserver started at http://127.19.185.193:34049/ using document root <none> and password file <none>
I20260926 02:55:49.617905 32675 fs_manager.cc:359] Metadata directory not provided
I20260926 02:55:49.618095 32675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:55:49.618484 32675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:55:49 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391349, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:55:49.642774 32675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "290e4d69bb1343dfaf7f50085105d280"
format_stamp: "Formatted at 2026-09-26 02:55:49 on dist-test-slave-ktng"
server_key: "7e1965b788917f5a773408d6c7450c8a"
server_key_iv: "d4ef2d09d979687e7b28eb580b83090e"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:49.644107 32675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "290e4d69bb1343dfaf7f50085105d280"
format_stamp: "Formatted at 2026-09-26 02:55:49 on dist-test-slave-ktng"
server_key: "7e1965b788917f5a773408d6c7450c8a"
server_key_iv: "d4ef2d09d979687e7b28eb580b83090e"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:49.658041 32675 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.007s
I20260926 02:55:49.686594 32695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:49.688642 32675 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.007s
I20260926 02:55:49.688921 32675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "290e4d69bb1343dfaf7f50085105d280"
format_stamp: "Formatted at 2026-09-26 02:55:49 on dist-test-slave-ktng"
server_key: "7e1965b788917f5a773408d6c7450c8a"
server_key_iv: "d4ef2d09d979687e7b28eb580b83090e"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:49.689240 32675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:55:49.725029 32675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:55:49.729290 32675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:55:49.729614 32675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:55:49.730703 32675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:55:49.732298 32675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:55:49.732420 32675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:49.732545 32675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:55:49.732631 32675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:49.820602 32675 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:40687
I20260926 02:55:49.820647 339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:40687 every 8 connection(s)
I20260926 02:55:49.822548 32675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:55:49.829224 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 32675
I20260926 02:55:49.829401 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:55:49.830174 20199 external_mini_cluster.cc:1428] Setting key 54334f9da2bb55705d1e22fced6f26a0
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
Sep 26 02:55:49 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391349, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:55:49.881173 343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:49.828145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:55845 (local address 127.19.185.254:41781)
0926 02:55:49.854164 (+ 26019us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:49.854175 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:49.854203 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:55:49.854278 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:49.854283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:49.854384 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:49.855640 (+ 1256us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:49.856914 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:49.858101 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:49.860632 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:49.861289 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:49.862339 (+ 1050us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:49.870864 (+ 8525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:49.870873 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:49.870879 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:49.870923 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:49.874939 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:49.878118 (+ 3179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:49.878124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:49.878128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:49.878286 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:49.878933 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:49.878939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:49.878942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:49.879348 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:49.879509 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:49.880543 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0926 02:55:49.880781 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25838,"thread_start_us":26383,"threads_started":1}
I20260926 02:55:49.883996 342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:49.827541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:41781 (local address 127.19.185.193:55845)
0926 02:55:49.842322 (+ 14781us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:49.842380 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:49.844939 (+ 2559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:49.855028 (+ 10089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:49.855044 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:49.855791 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:49.856685 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:49.856714 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:49.858302 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:49.858308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:49.860418 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:49.860436 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:49.861574 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:49.861583 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:49.861879 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:49.864060 (+ 2181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:49.864097 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:49.870528 (+ 6431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:49.875980 (+ 5452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:49.875991 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:49.876013 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:49.876661 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:49.878522 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:49.878527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:49.878531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:49.878797 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:49.879606 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:49.879615 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:49.880083 (+ 468us) client_negotiation.cc:769] Sending connection context
0926 02:55:49.881950 (+ 1867us) client_negotiation.cc:240] Negotiation successful
0926 02:55:49.882529 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14433,"thread_start_us":14189,"threads_started":1}
I20260926 02:55:49.886227 340 heartbeater.cc:347] Connected to a master server at 127.19.185.254:41781
I20260926 02:55:49.886566 340 heartbeater.cc:464] Registering TS with master...
I20260926 02:55:49.887537 340 heartbeater.cc:511] Master 127.19.185.254:41781 requested a full tablet report, sending...
I20260926 02:55:49.890272 32595 ts_manager.cc:194] Registered new tserver with Master: 290e4d69bb1343dfaf7f50085105d280 (127.19.185.193:40687)
I20260926 02:55:49.892407 32595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:55845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:55:49.904066 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.185.212:52161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:41781
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:55:49 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391349, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
Sep 26 02:55:50 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391349, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
W20260926 02:55:50.398288 346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:55:50.398803 346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:55:50.399012 346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:55:50.410566 346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:55:50.410797 346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:55:50.411036 346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:55:50.426632 346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.185.212:52161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:41781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:55:50.428742 346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:55:50.430608 346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:55:50.470551 353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:50.470598 354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:50.476948 346 server_base.cc:1034] running on GCE node
W20260926 02:55:50.477346 356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:50.478281 346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:55:50.480247 346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:55:50.481593 346 hybrid_clock.cc:648] HybridClock initialized: now 1790391350481498 us; error 60 us; skew 500 ppm
I20260926 02:55:50.482021 346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:55:50 dist-test-slave-ktng krb5kdc[32119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391350, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:55:50.486333 346 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:55:50.488878 346 webserver.cc:460] Webserver started at http://127.19.185.194:36445/ using document root <none> and password file <none>
I20260926 02:55:50.489893 346 fs_manager.cc:359] Metadata directory not provided
I20260926 02:55:50.490031 346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:55:50.490371 346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:55:50 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391350, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:55:50.509500 346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "3437e5c5400d4f0480e025ab6a174284"
format_stamp: "Formatted at 2026-09-26 02:55:50 on dist-test-slave-ktng"
server_key: "195d6993ff5cb74f4bb6224f662b1a57"
server_key_iv: "0c82f1eccd050f723645f3ba5fe10876"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:50.511111 346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "3437e5c5400d4f0480e025ab6a174284"
format_stamp: "Formatted at 2026-09-26 02:55:50 on dist-test-slave-ktng"
server_key: "195d6993ff5cb74f4bb6224f662b1a57"
server_key_iv: "0c82f1eccd050f723645f3ba5fe10876"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:50.537710 346 fs_manager.cc:691] Time spent creating directory manager: real 0.026s user 0.016s sys 0.008s
I20260926 02:55:50.568149 365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:50.570310 346 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.010s sys 0.005s
I20260926 02:55:50.570580 346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "3437e5c5400d4f0480e025ab6a174284"
format_stamp: "Formatted at 2026-09-26 02:55:50 on dist-test-slave-ktng"
server_key: "195d6993ff5cb74f4bb6224f662b1a57"
server_key_iv: "0c82f1eccd050f723645f3ba5fe10876"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:50.570946 346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:55:50.598119 346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:55:50.602311 346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:55:50.602640 346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:55:50.603729 346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:55:50.605134 346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:55:50.605239 346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:50.605393 346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:55:50.605469 346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:50.646317 346 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:36985
I20260926 02:55:50.646320 482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:36985 every 8 connection(s)
I20260926 02:55:50.647727 346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:55:50.652215 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 346
I20260926 02:55:50.652381 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:55:50.653117 20199 external_mini_cluster.cc:1428] Setting key 337743b9d5769d65619c08654c01307d
Sep 26 02:55:50 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391350, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:55:50.669742 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:50.650728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:34073 (local address 127.19.185.254:41781)
0926 02:55:50.651210 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:50.651218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:50.652434 (+ 1216us) server_negotiation.cc:407] Connection header received
0926 02:55:50.654501 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:50.654506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:50.654600 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:50.654723 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:50.656287 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:50.657020 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:50.658620 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:50.659014 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:50.659150 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:50.662425 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:50.662430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:50.662433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:50.662467 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:50.665762 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:50.666744 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:50.666750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:50.666753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:50.666913 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:50.667508 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:50.667514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:50.667518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:50.667926 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:50.668093 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:50.669054 (+ 961us) server_negotiation.cc:299] Negotiation successful
0926 02:55:50.669348 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":257,"threads_started":1}
I20260926 02:55:50.670915 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:50.651267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:41781 (local address 127.19.185.194:34073)
0926 02:55:50.652246 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:50.652309 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:50.654243 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:50.654951 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:50.654959 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:50.655555 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:50.656116 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:50.656133 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:50.657182 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:50.657187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:50.658459 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:50.658478 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:50.659232 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:50.659239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:50.659490 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:50.660170 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:50.660191 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:50.662206 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:50.665906 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:50.665918 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:50.665939 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:50.666577 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:50.667044 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:50.667057 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:50.667063 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:50.667330 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:50.668082 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:50.668093 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:50.668592 (+ 499us) client_negotiation.cc:769] Sending connection context
0926 02:55:50.668898 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 02:55:50.669479 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":422,"threads_started":1}
I20260926 02:55:50.672482 483 heartbeater.cc:347] Connected to a master server at 127.19.185.254:41781
I20260926 02:55:50.672832 483 heartbeater.cc:464] Registering TS with master...
I20260926 02:55:50.673696 483 heartbeater.cc:511] Master 127.19.185.254:41781 requested a full tablet report, sending...
I20260926 02:55:50.675521 32595 ts_manager.cc:194] Registered new tserver with Master: 3437e5c5400d4f0480e025ab6a174284 (127.19.185.194:36985)
I20260926 02:55:50.676728 32595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:34073
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:55:50.714274 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.185.212:52161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:41781
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:55:50.896345 340 heartbeater.cc:503] Master 127.19.185.254:41781 was elected leader, sending a full tablet report...
W20260926 02:55:51.055310 489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:55:51.055738 489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:55:51.055919 489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:55:51.063215 489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:55:51.063377 489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:55:51.063573 489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:55:51.073482 489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.185.212:52161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:41781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:55:51.074790 489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:55:51.076341 489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:55:51.089814 494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:51.089959 497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:51.090001 495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:51.090962 489 server_base.cc:1034] running on GCE node
I20260926 02:55:51.091636 489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:55:51.092585 489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:55:51.093779 489 hybrid_clock.cc:648] HybridClock initialized: now 1790391351093715 us; error 65 us; skew 500 ppm
I20260926 02:55:51.094098 489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:55:51 dist-test-slave-ktng krb5kdc[32119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391351, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:55:51.097555 489 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:55:51.099054 489 webserver.cc:460] Webserver started at http://127.19.185.195:41779/ using document root <none> and password file <none>
I20260926 02:55:51.099761 489 fs_manager.cc:359] Metadata directory not provided
I20260926 02:55:51.099860 489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:55:51.100112 489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:55:51 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391351, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:55:51.118667 489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "c80383aab1b54388b43b9a06b0291f08"
format_stamp: "Formatted at 2026-09-26 02:55:51 on dist-test-slave-ktng"
server_key: "d1c8ccd0574ca444605f00954a2605c3"
server_key_iv: "58420954de22cbacf1e684d3ff64b0d1"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:51.119894 489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "c80383aab1b54388b43b9a06b0291f08"
format_stamp: "Formatted at 2026-09-26 02:55:51 on dist-test-slave-ktng"
server_key: "d1c8ccd0574ca444605f00954a2605c3"
server_key_iv: "58420954de22cbacf1e684d3ff64b0d1"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:51.127115 489 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 02:55:51.144259 504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:51.146046 489 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:55:51.146239 489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "c80383aab1b54388b43b9a06b0291f08"
format_stamp: "Formatted at 2026-09-26 02:55:51 on dist-test-slave-ktng"
server_key: "d1c8ccd0574ca444605f00954a2605c3"
server_key_iv: "58420954de22cbacf1e684d3ff64b0d1"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:51.146497 489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:55:51.185176 489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:55:51.189081 489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:55:51.189334 489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:55:51.190248 489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:55:51.191614 489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:55:51.191686 489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:51.191771 489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:55:51.191816 489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:51.227113 489 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:43701
I20260926 02:55:51.227149 616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:43701 every 8 connection(s)
I20260926 02:55:51.228403 489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:55:51.229856 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 489
I20260926 02:55:51.230027 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:55:51.230468 20199 external_mini_cluster.cc:1428] Setting key fbe2e6fa7d668e6e4a752abf600c2fe9
Sep 26 02:55:51 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391351, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:55:51.249150 619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:51.231000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:60265 (local address 127.19.185.254:41781)
0926 02:55:51.231705 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:51.231709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:51.233755 (+ 2046us) server_negotiation.cc:407] Connection header received
0926 02:55:51.235287 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:51.235317 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:51.235412 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:51.235561 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:51.237235 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:51.237958 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:51.239579 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:51.240013 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:51.240149 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:51.243453 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:51.243458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:51.243461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:51.243492 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:51.246117 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:51.246837 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:51.246849 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:51.246852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:51.246971 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:51.247461 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:51.247466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:51.247469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:51.247748 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:51.247904 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:51.248569 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 02:55:51.248793 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":403,"threads_started":1}
I20260926 02:55:51.250064 620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:51.231704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:41781 (local address 127.19.185.195:60265)
0926 02:55:51.232562 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:51.232612 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:51.235010 (+ 2398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:51.235835 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:51.235846 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:51.236495 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:51.237073 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:51.237091 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:51.238112 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:51.238118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:51.239443 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:51.239455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:51.240234 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:51.240241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:51.240485 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:51.241175 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:51.241196 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:51.243204 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:51.246271 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:51.246280 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:51.246298 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:51.246712 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:51.247081 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:51.247087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:51.247091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:51.247338 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:51.247874 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:51.247882 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:51.248208 (+ 326us) client_negotiation.cc:769] Sending connection context
0926 02:55:51.248472 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 02:55:51.248968 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":305,"threads_started":1}
I20260926 02:55:51.251391 617 heartbeater.cc:347] Connected to a master server at 127.19.185.254:41781
I20260926 02:55:51.251731 617 heartbeater.cc:464] Registering TS with master...
I20260926 02:55:51.252562 617 heartbeater.cc:511] Master 127.19.185.254:41781 requested a full tablet report, sending...
I20260926 02:55:51.253960 32595 ts_manager.cc:194] Registered new tserver with Master: c80383aab1b54388b43b9a06b0291f08 (127.19.185.195:43701)
I20260926 02:55:51.254874 32595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:60265
I20260926 02:55:51.258719 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:55:51.260553 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 32675
Sep 26 02:55:51 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:55:51.277913 622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:51.261562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.19.185.193:40687)
0926 02:55:51.262249 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:51.262256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:51.262283 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:55:51.262355 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:51.262361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:51.262575 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:51.262708 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:51.264230 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:51.265288 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:51.266780 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:51.267408 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:51.267529 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:51.272210 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:51.272224 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:51.272239 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:51.272277 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:51.275019 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:51.275854 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:51.275866 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:51.275870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:51.276063 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:51.276420 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:51.276423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:51.276425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:51.276815 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:51.277088 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:51.277314 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 02:55:51.277530 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":378,"threads_started":1}
I20260926 02:55:51.279218 335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:55:51.362886 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 32675
I20260926 02:55:51.378901 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 346
Sep 26 02:55:51 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:55:51.395147 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:51.379871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.19.185.194:36985)
0926 02:55:51.380472 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:51.380479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:51.380505 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:55:51.380567 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:51.380573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:51.380786 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:51.380917 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:51.382195 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:51.383182 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:51.384519 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:51.385090 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:51.385194 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:51.389354 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:51.389366 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:51.389380 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:51.389412 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:51.392369 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:51.392966 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:51.392973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:51.392976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:51.393135 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:51.393443 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:51.393446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:51.393448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:51.393762 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:51.394151 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:51.394409 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 02:55:51.394738 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":336,"threads_started":1}
I20260926 02:55:51.396469 478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:55:51.478274 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 346
I20260926 02:55:51.498311 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 489
Sep 26 02:55:51 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:55:51.514638 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:51.499397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44888 (local address 127.19.185.195:43701)
0926 02:55:51.500058 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:51.500065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:51.500093 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:55:51.500154 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:51.500160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:51.500368 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:51.500501 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:51.501871 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:51.502879 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:51.504411 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:51.504957 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:51.505056 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:51.509293 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:51.509304 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:51.509320 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:51.509354 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:51.512199 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:51.512826 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:51.512834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:51.512836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:51.512953 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:51.513256 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:51.513259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:51.513261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:51.513585 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:51.513815 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:51.514027 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 02:55:51.514211 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":344,"threads_started":1}
I20260926 02:55:51.516036 612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:55:51.601183 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 489
I20260926 02:55:51.617313 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 32563
I20260926 02:55:51.618611 32625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:55:51.724969 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 32563
I20260926 02:55:51.751464 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:41781
--webserver_interface=127.19.185.254
--webserver_port=37309
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:41781
--ranger_config_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.185.212:52161/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.185.212:52161/kms with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:55:52.074018 629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:55:52.074302 629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:55:52.074478 629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:55:52.081882 629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:55:52.082020 629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:55:52.082135 629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:55:52.082230 629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:55:52.082316 629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:55:52.092573 629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.185.212:52161/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:41781
--ranger_config_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:41781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=37309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:55:52.093967 629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:55:52.095381 629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:55:52.107667 635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:52.107667 637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:52.108382 634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:52.108495 629 server_base.cc:1034] running on GCE node
I20260926 02:55:52.109272 629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:55:52.110569 629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:55:52.111727 629 hybrid_clock.cc:648] HybridClock initialized: now 1790391352111679 us; error 44 us; skew 500 ppm
I20260926 02:55:52.112185 629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:55:52 dist-test-slave-ktng krb5kdc[32119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391352, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:55:52.115492 629 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:55:52.117062 629 webserver.cc:460] Webserver started at http://127.19.185.254:37309/ using document root <none> and password file <none>
I20260926 02:55:52.117767 629 fs_manager.cc:359] Metadata directory not provided
I20260926 02:55:52.117951 629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:55:52 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391352, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:55:52.172027 629 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:55:52.175081 645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:52.176553 629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:55:52.176759 629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc"
format_stamp: "Formatted at 2026-09-26 02:55:48 on dist-test-slave-ktng"
server_key: "3cefbdf53e711a41161ed28e131edcd3"
server_key_iv: "b3a2cb41ab80e28b5e61d2645565a781"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:52.177775 629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:55:52.192497 629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:55:52.196247 629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:55:52.196542 629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:55:52.215492 629 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:41781
I20260926 02:55:52.215471 696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:41781 every 8 connection(s)
I20260926 02:55:52.217043 629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:55:52.223850 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 629
I20260926 02:55:52.224170 697 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:55:52.224738 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.185.212:52161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:40687
--local_ip_for_outbound_sockets=127.19.185.193
--tserver_master_addrs=127.19.185.254:41781
--webserver_port=34049
--webserver_interface=127.19.185.193
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.185.212:52161/kms with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:55:52.227494 697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc: Bootstrap starting.
I20260926 02:55:52.247452 697 log.cc:824] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc: Log is configured to *not* fsync() on all Append() calls
I20260926 02:55:52.256217 697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:55:52.256759 697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc: Bootstrap complete.
I20260926 02:55:52.265676 697 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41781 } }
I20260926 02:55:52.266487 697 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7eda3c2b2ae44b0bb5ccf46f4c740fc, State: Initialized, Role: FOLLOWER
I20260926 02:55:52.267125 697 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41781 } }
I20260926 02:55:52.267385 697 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:55:52.267518 697 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:55:52.267696 697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:55:52.271417 697 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41781 } }
I20260926 02:55:52.271924 697 leader_election.cc:302] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7eda3c2b2ae44b0bb5ccf46f4c740fc; no voters:
I20260926 02:55:52.272570 697 leader_election.cc:288] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:55:52.272727 701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:55:52.273893 701 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [term 2 LEADER]: Becoming Leader. State: Replica: b7eda3c2b2ae44b0bb5ccf46f4c740fc, State: Running, Role: LEADER
I20260926 02:55:52.274509 701 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41781 } }
I20260926 02:55:52.274792 697 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:55:52.277484 703 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7eda3c2b2ae44b0bb5ccf46f4c740fc. Latest consensus state: current_term: 2 leader_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41781 } } }
I20260926 02:55:52.277846 703 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [sys.catalog]: This master's current role is: LEADER
I20260926 02:55:52.277526 702 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eda3c2b2ae44b0bb5ccf46f4c740fc" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41781 } } }
I20260926 02:55:52.278055 702 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc [sys.catalog]: This master's current role is: LEADER
I20260926 02:55:52.286725 707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:55:52.293699 707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:55:52.294719 697 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 02:55:52.295075 707 catalog_manager.cc:1259] Loaded cluster ID: ccc3f88e210740f1b9ca8b87ae0db489
I20260926 02:55:52.295244 707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:55:52.299461 707 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:55:52.301744 707 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc: Loaded TSK: 0
I20260926 02:55:52.303339 707 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:55:52.646545 699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:55:52.647069 699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:55:52.647289 699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:55:52.659271 699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:55:52.659538 699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:55:52.659792 699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:55:52.676131 699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.185.212:52161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:40687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=34049
--tserver_master_addrs=127.19.185.254:41781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:55:52.678148 699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:55:52.680016 699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:55:52.711602 734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:52.712553 735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:52.720963 699 server_base.cc:1034] running on GCE node
W20260926 02:55:52.723465 737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:52.724514 699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:55:52.729525 699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:55:52.733008 699 hybrid_clock.cc:648] HybridClock initialized: now 1790391352732923 us; error 74 us; skew 500 ppm
I20260926 02:55:52.733552 699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:55:52 dist-test-slave-ktng krb5kdc[32119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391352, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:55:52.738114 699 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:55:52.740243 699 webserver.cc:460] Webserver started at http://127.19.185.193:34049/ using document root <none> and password file <none>
I20260926 02:55:52.741241 699 fs_manager.cc:359] Metadata directory not provided
I20260926 02:55:52.741402 699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:55:52 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391352, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:55:52.801124 699 fs_manager.cc:709] Time spent opening directory manager: real 0.025s user 0.011s sys 0.008s
I20260926 02:55:52.811214 745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:52.813346 699 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.009s sys 0.000s
I20260926 02:55:52.813647 699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "290e4d69bb1343dfaf7f50085105d280"
format_stamp: "Formatted at 2026-09-26 02:55:49 on dist-test-slave-ktng"
server_key: "7e1965b788917f5a773408d6c7450c8a"
server_key_iv: "d4ef2d09d979687e7b28eb580b83090e"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:52.815327 699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:55:52.893473 699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:55:52.899283 699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:55:52.899726 699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:55:52.907284 699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:55:52.919207 699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:55:52.919466 699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 02:55:52.919672 699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:55:52.919827 699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:53.062454 699 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:40687
I20260926 02:55:53.062516 858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:40687 every 8 connection(s)
I20260926 02:55:53.064678 699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:55:53.074425 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 699
I20260926 02:55:53.075320 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.185.212:52161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:36985
--local_ip_for_outbound_sockets=127.19.185.194
--tserver_master_addrs=127.19.185.254:41781
--webserver_port=36445
--webserver_interface=127.19.185.194
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.185.212:52161/kms with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:55:53 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391352, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:55:53.126096 862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:53.070390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:42457 (local address 127.19.185.254:41781)
0926 02:55:53.091871 (+ 21481us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:53.091888 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:53.091956 (+ 68us) server_negotiation.cc:407] Connection header received
0926 02:55:53.092458 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:53.092486 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:53.093076 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:53.095842 (+ 2766us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:53.102305 (+ 6463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:53.103965 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:53.106576 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:53.107750 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:53.107972 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:53.113071 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:53.113082 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:53.113104 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:53.113167 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:53.117963 (+ 4796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:53.119042 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:53.119051 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:53.119059 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:53.119278 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:53.119889 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:53.119894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:53.119897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:53.120585 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:53.120771 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:53.124308 (+ 3537us) server_negotiation.cc:299] Negotiation successful
0926 02:55:53.124756 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21201,"thread_start_us":20974,"threads_started":1}
I20260926 02:55:53.123908 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:53.068531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:41781 (local address 127.19.185.193:42457)
0926 02:55:53.074844 (+ 6313us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:53.074911 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:53.077989 (+ 3078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:53.096032 (+ 18043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:53.096046 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:53.096793 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:53.097739 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:53.097803 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:53.104181 (+ 6378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:53.104189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:53.106346 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:53.106368 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:53.108423 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:53.108433 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:53.108723 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:53.109687 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:53.109722 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:53.112722 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:53.118167 (+ 5445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:53.118178 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:53.118196 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:53.118837 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:53.119455 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:53.119460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:53.119463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:53.119721 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:53.120881 (+ 1160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:53.120890 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:53.121351 (+ 461us) client_negotiation.cc:769] Sending connection context
0926 02:55:53.121694 (+ 343us) client_negotiation.cc:240] Negotiation successful
0926 02:55:53.122350 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5965,"thread_start_us":5733,"threads_started":1}
I20260926 02:55:53.131170 859 heartbeater.cc:347] Connected to a master server at 127.19.185.254:41781
I20260926 02:55:53.131603 859 heartbeater.cc:464] Registering TS with master...
I20260926 02:55:53.132483 859 heartbeater.cc:511] Master 127.19.185.254:41781 requested a full tablet report, sending...
W20260926 02:55:53.135842 859 heartbeater.cc:644] Failed to heartbeat to 127.19.185.254:41781 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:55:53.136096 859 heartbeater.cc:464] Registering TS with master...
I20260926 02:55:53.136384 859 heartbeater.cc:511] Master 127.19.185.254:41781 requested a full tablet report, sending...
I20260926 02:55:53.137748 859 heartbeater.cc:464] Registering TS with master...
I20260926 02:55:53.138118 859 heartbeater.cc:511] Master 127.19.185.254:41781 requested a full tablet report, sending...
W20260926 02:55:53.139289 859 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:55:53.513845 863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:55:53.514171 863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:55:53.514343 863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:55:53.522089 863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:55:53.522256 863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:55:53.522436 863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:55:53.533938 863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.185.212:52161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:36985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=36445
--tserver_master_addrs=127.19.185.254:41781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:55:53.535480 863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:55:53.536948 863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:55:53.550851 876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:53.551390 875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:53.551771 878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:53.552662 863 server_base.cc:1034] running on GCE node
I20260926 02:55:53.553495 863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:55:53.554687 863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:55:53.556011 863 hybrid_clock.cc:648] HybridClock initialized: now 1790391353555930 us; error 56 us; skew 500 ppm
I20260926 02:55:53.556391 863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:55:53 dist-test-slave-ktng krb5kdc[32119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391353, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:55:53.560192 863 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:55:53.561928 863 webserver.cc:460] Webserver started at http://127.19.185.194:36445/ using document root <none> and password file <none>
I20260926 02:55:53.562641 863 fs_manager.cc:359] Metadata directory not provided
I20260926 02:55:53.562810 863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:55:53 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391353, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:55:53.592185 863 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.007s sys 0.000s
I20260926 02:55:53.595955 888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:53.597874 863 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:55:53.598064 863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "3437e5c5400d4f0480e025ab6a174284"
format_stamp: "Formatted at 2026-09-26 02:55:50 on dist-test-slave-ktng"
server_key: "195d6993ff5cb74f4bb6224f662b1a57"
server_key_iv: "0c82f1eccd050f723645f3ba5fe10876"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:53.599488 863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:55:53.614087 863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:55:53.618479 863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:55:53.618794 863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:55:53.619998 863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:55:53.636199 863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:55:53.636385 863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 02:55:53.636538 863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:55:53.636647 863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:53.729045 863 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:36985
I20260926 02:55:53.729125 1002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:36985 every 8 connection(s)
I20260926 02:55:53.731016 863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:55:53.739141 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 863
I20260926 02:55:53.740164 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.185.212:52161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:43701
--local_ip_for_outbound_sockets=127.19.185.195
--tserver_master_addrs=127.19.185.254:41781
--webserver_port=41779
--webserver_interface=127.19.185.195
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.185.212:52161/kms with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:55:53 dist-test-slave-ktng krb5kdc[32119](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790391353, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 02:55:53 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391353, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:55:53.784076 1005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:53.735634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:43539 (local address 127.19.185.254:41781)
0926 02:55:53.747766 (+ 12132us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:53.747776 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:53.748963 (+ 1187us) server_negotiation.cc:407] Connection header received
0926 02:55:53.751498 (+ 2535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:53.751515 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:53.751623 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:53.751709 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:53.754120 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:53.755385 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:53.757720 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:53.758419 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:53.758527 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:53.766272 (+ 7745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:53.766280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:53.766286 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:53.766333 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:53.770071 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:53.780550 (+ 10479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:53.780557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:53.780560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:53.780817 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:53.781463 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:53.781480 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:53.781484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:53.782016 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:53.782166 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:53.783355 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0926 02:55:53.783684 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11943,"thread_start_us":11818,"threads_started":1}
I20260926 02:55:53.786715 1006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:53.736778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:41781 (local address 127.19.185.194:43539)
0926 02:55:53.748415 (+ 11637us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:53.748475 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:53.751108 (+ 2633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:53.752294 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:53.752308 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:53.753013 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:53.753867 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:53.753899 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:53.755594 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:53.755602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:53.757541 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:53.757556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:53.758909 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:53.758919 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:53.759221 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:53.762228 (+ 3007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:53.762254 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:53.765921 (+ 3667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:53.779531 (+ 13610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:53.779543 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:53.779565 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:53.780330 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:53.781009 (+ 679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:53.781016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:53.781020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:53.781319 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:53.782273 (+ 954us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:53.782282 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:53.782880 (+ 598us) client_negotiation.cc:769] Sending connection context
0926 02:55:53.784644 (+ 1764us) client_negotiation.cc:240] Negotiation successful
0926 02:55:53.785268 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11323,"thread_start_us":11089,"threads_started":1}
I20260926 02:55:53.791952 1003 heartbeater.cc:347] Connected to a master server at 127.19.185.254:41781
I20260926 02:55:53.792301 1003 heartbeater.cc:464] Registering TS with master...
I20260926 02:55:53.793215 1003 heartbeater.cc:511] Master 127.19.185.254:41781 requested a full tablet report, sending...
I20260926 02:55:53.795365 662 ts_manager.cc:194] Registered new tserver with Master: 3437e5c5400d4f0480e025ab6a174284 (127.19.185.194:36985)
I20260926 02:55:53.797559 662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:43539
I20260926 02:55:54.139475 859 heartbeater.cc:464] Registering TS with master...
I20260926 02:55:54.139923 859 heartbeater.cc:511] Master 127.19.185.254:41781 requested a full tablet report, sending...
I20260926 02:55:54.141363 662 ts_manager.cc:194] Registered new tserver with Master: 290e4d69bb1343dfaf7f50085105d280 (127.19.185.193:40687)
I20260926 02:55:54.142542 662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:42457
W20260926 02:55:54.218400 1007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:55:54.218854 1007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:55:54.219055 1007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:55:54.230747 1007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:55:54.230983 1007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:55:54.231225 1007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:55:54.243068 1007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.185.212:52161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:43701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=41779
--tserver_master_addrs=127.19.185.254:41781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
Sep 26 02:55:54 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391353, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:55:54.244578 1007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:55:54.246336 1007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:55:54.299469 1014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:54.300055 1013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:54.308122 1016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:54.311512 1007 server_base.cc:1034] running on GCE node
I20260926 02:55:54.312389 1007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:55:54.323628 1007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:55:54.324983 1007 hybrid_clock.cc:648] HybridClock initialized: now 1790391354324909 us; error 76 us; skew 500 ppm
I20260926 02:55:54.325410 1007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:55:54 dist-test-slave-ktng krb5kdc[32119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391354, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:55:54.328807 1007 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:55:54.330686 1007 webserver.cc:460] Webserver started at http://127.19.185.195:41779/ using document root <none> and password file <none>
I20260926 02:55:54.331519 1007 fs_manager.cc:359] Metadata directory not provided
I20260926 02:55:54.331677 1007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:55:54 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391354, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:55:54.354595 1007 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:55:54.358107 1024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:54.359694 1007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:55:54.359876 1007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "c80383aab1b54388b43b9a06b0291f08"
format_stamp: "Formatted at 2026-09-26 02:55:51 on dist-test-slave-ktng"
server_key: "d1c8ccd0574ca444605f00954a2605c3"
server_key_iv: "58420954de22cbacf1e684d3ff64b0d1"
server_key_version: "kuduclusterkey@0"
I20260926 02:55:54.360886 1007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:55:54.389590 1007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:55:54.394829 1007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:55:54.395156 1007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:55:54.396806 1007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:55:54.398538 1007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:55:54.398649 1007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:54.398752 1007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:55:54.398826 1007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:54.446424 1007 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:43701
I20260926 02:55:54.446484 1136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:43701 every 8 connection(s)
I20260926 02:55:54.448424 1007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:55:54.453114 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 1007
Sep 26 02:55:54 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391354, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:55:54.498222 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.454898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35932 (local address 127.19.185.254:41781)
0926 02:55:54.477997 (+ 23099us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:54.478003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:54.478019 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:55:54.478082 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:54.478087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:54.478177 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:54.478294 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:54.480961 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.482255 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.484613 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.485360 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.485481 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.489431 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:54.489437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:54.489442 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:54.489480 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:54.493738 (+ 4258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:54.494545 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:54.494550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:54.494553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:54.494984 (+ 431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:54.495472 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:54.495476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:54.495479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:54.496082 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:54.497465 (+ 1383us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:54.497606 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 02:55:54.497838 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":22966,"thread_start_us":16549,"threads_started":1}
I20260926 02:55:54.503823 1142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.451453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:40753 (local address 127.19.185.254:41781)
0926 02:55:54.471764 (+ 20311us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:54.471774 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:54.477579 (+ 5805us) server_negotiation.cc:407] Connection header received
0926 02:55:54.480854 (+ 3275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:54.480860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:54.480962 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:54.481096 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:54.484643 (+ 3547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.485939 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.488416 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.489051 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.489152 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.494064 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:54.494071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:54.494077 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:54.494116 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:54.499479 (+ 5363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:54.500575 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:54.500581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:54.500585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:54.500712 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:54.501637 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:54.501642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:54.501645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:54.502016 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:54.502154 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:54.503184 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0926 02:55:54.503431 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":20136,"thread_start_us":26263,"threads_started":1}
I20260926 02:55:54.506413 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.451918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:41781 (local address 127.19.185.195:40753)
0926 02:55:54.477394 (+ 25476us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:54.477454 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:54.480542 (+ 3088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:54.481706 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:54.481719 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:54.483420 (+ 1701us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:54.484405 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:54.484433 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.486151 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.486157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:54.488243 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:54.488258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.489518 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.489526 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:54.489810 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.490634 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:54.490671 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:54.493749 (+ 3078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:54.499656 (+ 5907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:54.499666 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:54.499688 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:54.500438 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:54.500850 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:54.500855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:54.500859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:54.501134 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:54.502250 (+ 1116us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:54.502259 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:54.502723 (+ 464us) client_negotiation.cc:769] Sending connection context
0926 02:55:54.504091 (+ 1368us) client_negotiation.cc:240] Negotiation successful
0926 02:55:54.504787 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25174,"thread_start_us":24962,"threads_started":1}
I20260926 02:55:54.508153 1137 heartbeater.cc:347] Connected to a master server at 127.19.185.254:41781
I20260926 02:55:54.508496 1137 heartbeater.cc:464] Registering TS with master...
I20260926 02:55:54.509404 1137 heartbeater.cc:511] Master 127.19.185.254:41781 requested a full tablet report, sending...
I20260926 02:55:54.511183 662 ts_manager.cc:194] Registered new tserver with Master: c80383aab1b54388b43b9a06b0291f08 (127.19.185.195:43701)
I20260926 02:55:54.512356 662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:40753
I20260926 02:55:54.515745 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:55:54.545820 1142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.529304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.19.185.254:41781)
0926 02:55:54.529499 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:54.529505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:54.529614 (+ 109us) server_negotiation.cc:407] Connection header received
0926 02:55:54.529887 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:54.529892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:54.529980 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:54.530063 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:54.532157 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.533204 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.535066 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.535701 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.535841 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.539092 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:54.539099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:54.539104 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:54.539149 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:54.542655 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:54.543486 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:54.543492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:54.543495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:54.543637 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:54.544083 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:54.544088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:54.544090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:54.544425 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:54.544620 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:54.545229 (+ 609us) server_negotiation.cc:299] Negotiation successful
0926 02:55:54.545469 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:55:54.551980 662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:55:54.563181 662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 26 02:55:54 dist-test-slave-ktng krb5kdc[32119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391353, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for HTTP/127.19.185.212@KRBTEST.COM
I20260926 02:55:54.616876 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.583955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:36985 (local address 127.0.0.1:37886)
0926 02:55:54.591920 (+ 7965us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:54.591994 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:54.592182 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:54.593697 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:54.593705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:54.593744 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:55:54.594215 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:54.594250 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.598940 (+ 4690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.598949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:54.602217 (+ 3268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:54.602239 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.615469 (+ 13230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.615479 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:54.615834 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.615914 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 02:55:54.616080 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 02:55:54.616362 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7667,"thread_start_us":7819,"threads_started":1}
I20260926 02:55:54.620121 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.584095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40687 (local address 127.0.0.1:42198)
0926 02:55:54.595484 (+ 11389us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:54.595512 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:54.595624 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:54.602417 (+ 6793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:54.602425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:54.602465 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:55:54.602882 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:54.602895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.609718 (+ 6823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.609727 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:54.612376 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:54.612400 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.619282 (+ 6882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.619290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:54.619504 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.619545 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 02:55:54.619626 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 02:55:54.619770 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11254,"thread_start_us":10795,"threads_started":1}
I20260926 02:55:54.621007 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.584695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.19.185.194:36985)
0926 02:55:54.592762 (+ 8067us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:54.592771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:54.592803 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:55:54.592911 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:54.592918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:54.593198 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:54.593396 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:55:54.596701 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.598745 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.612641 (+ 13896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.614688 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.614866 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.615085 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:54.620307 (+ 5222us) server_negotiation.cc:299] Negotiation successful
0926 02:55:54.620514 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7908,"thread_start_us":7764,"threads_started":1}
I20260926 02:55:54.625578 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.585169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42198 (local address 127.19.185.193:40687)
0926 02:55:54.595002 (+ 9833us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:54.595014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:54.595749 (+ 735us) server_negotiation.cc:407] Connection header received
0926 02:55:54.595867 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:54.595876 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:54.596158 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:54.596381 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:55:54.607440 (+ 11059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.609497 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.615196 (+ 5699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.618764 (+ 3568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.618939 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.619127 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:54.625000 (+ 5873us) server_negotiation.cc:299] Negotiation successful
0926 02:55:54.625186 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9615,"thread_start_us":9424,"threads_started":1}
I20260926 02:55:54.625522 938 tablet_service.cc:1467] Processing CreateTablet for tablet 77dc61ea73c14a5eb3e4fd1fb5cf092d (DEFAULT_TABLE table=test-table [id=fa98a092780e4096b6ec2e60277a41c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:55:54.627507 938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77dc61ea73c14a5eb3e4fd1fb5cf092d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:55:54.631866 794 tablet_service.cc:1467] Processing CreateTablet for tablet 77dc61ea73c14a5eb3e4fd1fb5cf092d (DEFAULT_TABLE table=test-table [id=fa98a092780e4096b6ec2e60277a41c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:55:54.633375 794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77dc61ea73c14a5eb3e4fd1fb5cf092d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:55:54.636322 1158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.598298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:43701 (local address 127.0.0.1:44898)
0926 02:55:54.614953 (+ 16655us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:54.614974 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:54.615106 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:54.618291 (+ 3185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:54.618296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:54.618323 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:55:54.618627 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:54.618639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.620795 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.620800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:54.633066 (+ 12266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:54.633096 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.635517 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.635524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:54.635702 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.635744 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 02:55:54.635835 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 02:55:54.635981 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16526,"thread_start_us":16894,"threads_started":1}
I20260926 02:55:54.646983 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.596010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.19.185.195:43701)
0926 02:55:54.617264 (+ 21254us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:54.617274 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:54.617311 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:55:54.617415 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:54.617424 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:54.617727 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:54.617967 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:55:54.619064 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.620648 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.633387 (+ 12739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.634964 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.635172 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.635361 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:54.646456 (+ 11095us) server_negotiation.cc:299] Negotiation successful
0926 02:55:54.646646 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21060,"thread_start_us":22759,"threads_started":1}
I20260926 02:55:54.650897 1072 tablet_service.cc:1467] Processing CreateTablet for tablet 77dc61ea73c14a5eb3e4fd1fb5cf092d (DEFAULT_TABLE table=test-table [id=fa98a092780e4096b6ec2e60277a41c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:55:54.652436 1072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77dc61ea73c14a5eb3e4fd1fb5cf092d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:55:54.666069 1159 tablet_bootstrap.cc:492] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284: Bootstrap starting.
I20260926 02:55:54.672529 1159 tablet_bootstrap.cc:654] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284: Neither blocks nor log segments found. Creating new log.
I20260926 02:55:54.673403 1160 tablet_bootstrap.cc:492] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280: Bootstrap starting.
I20260926 02:55:54.674728 1161 tablet_bootstrap.cc:492] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08: Bootstrap starting.
I20260926 02:55:54.677327 1160 tablet_bootstrap.cc:654] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280: Neither blocks nor log segments found. Creating new log.
I20260926 02:55:54.678555 1160 log.cc:824] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280: Log is configured to *not* fsync() on all Append() calls
I20260926 02:55:54.679356 1161 tablet_bootstrap.cc:654] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08: Neither blocks nor log segments found. Creating new log.
I20260926 02:55:54.680398 1159 log.cc:824] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284: Log is configured to *not* fsync() on all Append() calls
I20260926 02:55:54.684213 1161 log.cc:824] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08: Log is configured to *not* fsync() on all Append() calls
I20260926 02:55:54.687320 1159 tablet_bootstrap.cc:492] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284: No bootstrap required, opened a new log
I20260926 02:55:54.687675 1159 ts_tablet_manager.cc:1397] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.005s
I20260926 02:55:54.703495 1160 tablet_bootstrap.cc:492] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280: No bootstrap required, opened a new log
I20260926 02:55:54.703943 1160 ts_tablet_manager.cc:1397] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.011s
I20260926 02:55:54.713188 1161 tablet_bootstrap.cc:492] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08: No bootstrap required, opened a new log
I20260926 02:55:54.713603 1161 ts_tablet_manager.cc:1397] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.016s
I20260926 02:55:54.716807 1160 raft_consensus.cc:348] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e4d69bb1343dfaf7f50085105d280" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40687 } } peers { permanent_uuid: "c80383aab1b54388b43b9a06b0291f08" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43701 } } peers { permanent_uuid: "3437e5c5400d4f0480e025ab6a174284" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 36985 } }
I20260926 02:55:54.717612 1159 raft_consensus.cc:348] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e4d69bb1343dfaf7f50085105d280" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40687 } } peers { permanent_uuid: "c80383aab1b54388b43b9a06b0291f08" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43701 } } peers { permanent_uuid: "3437e5c5400d4f0480e025ab6a174284" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 36985 } }
I20260926 02:55:54.718128 1159 raft_consensus.cc:374] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:55:54.718247 1159 raft_consensus.cc:729] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3437e5c5400d4f0480e025ab6a174284, State: Initialized, Role: FOLLOWER
I20260926 02:55:54.719030 1159 consensus_queue.cc:260] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e4d69bb1343dfaf7f50085105d280" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40687 } } peers { permanent_uuid: "c80383aab1b54388b43b9a06b0291f08" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43701 } } peers { permanent_uuid: "3437e5c5400d4f0480e025ab6a174284" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 36985 } }
I20260926 02:55:54.721004 1160 raft_consensus.cc:374] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:55:54.721330 1160 raft_consensus.cc:729] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290e4d69bb1343dfaf7f50085105d280, State: Initialized, Role: FOLLOWER
I20260926 02:55:54.722271 1160 consensus_queue.cc:260] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e4d69bb1343dfaf7f50085105d280" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40687 } } peers { permanent_uuid: "c80383aab1b54388b43b9a06b0291f08" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43701 } } peers { permanent_uuid: "3437e5c5400d4f0480e025ab6a174284" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 36985 } }
I20260926 02:55:54.725235 1003 heartbeater.cc:503] Master 127.19.185.254:41781 was elected leader, sending a full tablet report...
I20260926 02:55:54.726269 1159 ts_tablet_manager.cc:1428] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284: Time spent starting tablet: real 0.038s user 0.011s sys 0.004s
I20260926 02:55:54.727085 859 heartbeater.cc:503] Master 127.19.185.254:41781 was elected leader, sending a full tablet report...
I20260926 02:55:54.730257 1160 ts_tablet_manager.cc:1428] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280: Time spent starting tablet: real 0.026s user 0.010s sys 0.002s
I20260926 02:55:54.739840 1161 raft_consensus.cc:348] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e4d69bb1343dfaf7f50085105d280" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40687 } } peers { permanent_uuid: "c80383aab1b54388b43b9a06b0291f08" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43701 } } peers { permanent_uuid: "3437e5c5400d4f0480e025ab6a174284" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 36985 } }
I20260926 02:55:54.742244 1161 raft_consensus.cc:374] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:55:54.742437 1161 raft_consensus.cc:729] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80383aab1b54388b43b9a06b0291f08, State: Initialized, Role: FOLLOWER
I20260926 02:55:54.743958 1161 consensus_queue.cc:260] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e4d69bb1343dfaf7f50085105d280" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40687 } } peers { permanent_uuid: "c80383aab1b54388b43b9a06b0291f08" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43701 } } peers { permanent_uuid: "3437e5c5400d4f0480e025ab6a174284" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 36985 } }
I20260926 02:55:54.746568 1137 heartbeater.cc:503] Master 127.19.185.254:41781 was elected leader, sending a full tablet report...
I20260926 02:55:54.749166 1161 ts_tablet_manager.cc:1428] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08: Time spent starting tablet: real 0.035s user 0.014s sys 0.000s
W20260926 02:55:54.750494 1004 tablet.cc:2369] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:55:54.785238 1165 raft_consensus.cc:482] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:55:54.785583 1165 raft_consensus.cc:504] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e4d69bb1343dfaf7f50085105d280" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40687 } } peers { permanent_uuid: "c80383aab1b54388b43b9a06b0291f08" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43701 } } peers { permanent_uuid: "3437e5c5400d4f0480e025ab6a174284" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 36985 } }
I20260926 02:55:54.787854 1165 leader_election.cc:288] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 290e4d69bb1343dfaf7f50085105d280 (127.19.185.193:40687), c80383aab1b54388b43b9a06b0291f08 (127.19.185.195:43701)
I20260926 02:55:54.809800 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.789384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:35817 (local address 127.19.185.193:40687)
0926 02:55:54.789632 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:54.789642 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:54.799950 (+ 10308us) server_negotiation.cc:407] Connection header received
0926 02:55:54.801675 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:54.801684 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:54.801772 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:54.801839 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:55:54.803223 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.804495 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.807370 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.808575 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.808712 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.808789 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:54.809394 (+ 605us) server_negotiation.cc:299] Negotiation successful
0926 02:55:54.809537 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 02:55:54.811100 1168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.788605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40687 (local address 127.19.185.194:35817)
0926 02:55:54.799695 (+ 11090us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:54.799720 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:54.801331 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:54.802081 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:54.802086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:54.802116 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:55:54.802455 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:54.802467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.804906 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.804911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:54.807159 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:54.807174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.808984 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.808990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:54.809192 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.809228 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:55:54.810481 (+ 1253us) client_negotiation.cc:240] Negotiation successful
0926 02:55:54.810635 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10932,"thread_start_us":11379,"threads_started":1}
I20260926 02:55:54.812554 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc61ea73c14a5eb3e4fd1fb5cf092d" candidate_uuid: "3437e5c5400d4f0480e025ab6a174284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290e4d69bb1343dfaf7f50085105d280" is_pre_election: true
I20260926 02:55:54.813122 814 raft_consensus.cc:2454] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3437e5c5400d4f0480e025ab6a174284 in term 0.
I20260926 02:55:54.814051 890 leader_election.cc:302] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290e4d69bb1343dfaf7f50085105d280, 3437e5c5400d4f0480e025ab6a174284; no voters:
I20260926 02:55:54.814651 1165 raft_consensus.cc:2781] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:55:54.814795 1165 raft_consensus.cc:482] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:55:54.814898 1165 raft_consensus.cc:3037] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:55:54.817245 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.800705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:38663 (local address 127.19.185.195:43701)
0926 02:55:54.800900 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:54.800906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:54.803750 (+ 2844us) server_negotiation.cc:407] Connection header received
0926 02:55:54.804008 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:54.804014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:54.804085 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:54.804144 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:55:54.804892 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.805987 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.814684 (+ 8697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.815972 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.816106 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.816189 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:54.816869 (+ 680us) server_negotiation.cc:299] Negotiation successful
0926 02:55:54.817004 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:55:54.818234 1169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.801059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:43701 (local address 127.19.185.194:38663)
0926 02:55:54.803430 (+ 2371us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:54.803449 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:54.803856 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:54.804312 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:54.804316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:54.804338 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:55:54.804718 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:54.804730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.810704 (+ 5974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.810710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:54.814430 (+ 3720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:54.814444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.816453 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.816460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:54.816666 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.816707 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 02:55:54.817460 (+ 753us) client_negotiation.cc:240] Negotiation successful
0926 02:55:54.817616 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2252,"thread_start_us":3511,"threads_started":1}
I20260926 02:55:54.818361 1165 raft_consensus.cc:504] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e4d69bb1343dfaf7f50085105d280" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40687 } } peers { permanent_uuid: "c80383aab1b54388b43b9a06b0291f08" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43701 } } peers { permanent_uuid: "3437e5c5400d4f0480e025ab6a174284" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 36985 } }
I20260926 02:55:54.818919 1165 leader_election.cc:288] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [CANDIDATE]: Term 1 election: Requested vote from peers 290e4d69bb1343dfaf7f50085105d280 (127.19.185.193:40687), c80383aab1b54388b43b9a06b0291f08 (127.19.185.195:43701)
I20260926 02:55:54.819967 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc61ea73c14a5eb3e4fd1fb5cf092d" candidate_uuid: "3437e5c5400d4f0480e025ab6a174284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290e4d69bb1343dfaf7f50085105d280"
I20260926 02:55:54.820187 814 raft_consensus.cc:3037] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:55:54.820506 1092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc61ea73c14a5eb3e4fd1fb5cf092d" candidate_uuid: "3437e5c5400d4f0480e025ab6a174284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c80383aab1b54388b43b9a06b0291f08" is_pre_election: true
I20260926 02:55:54.821072 1092 raft_consensus.cc:2454] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3437e5c5400d4f0480e025ab6a174284 in term 0.
I20260926 02:55:54.821693 1091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc61ea73c14a5eb3e4fd1fb5cf092d" candidate_uuid: "3437e5c5400d4f0480e025ab6a174284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c80383aab1b54388b43b9a06b0291f08"
I20260926 02:55:54.821928 1091 raft_consensus.cc:3037] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:55:54.822990 814 raft_consensus.cc:2454] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3437e5c5400d4f0480e025ab6a174284 in term 1.
I20260926 02:55:54.823635 890 leader_election.cc:302] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290e4d69bb1343dfaf7f50085105d280, 3437e5c5400d4f0480e025ab6a174284; no voters:
I20260926 02:55:54.823993 1165 raft_consensus.cc:2781] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:55:54.825047 1091 raft_consensus.cc:2454] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3437e5c5400d4f0480e025ab6a174284 in term 1.
I20260926 02:55:54.835189 1165 raft_consensus.cc:686] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [term 1 LEADER]: Becoming Leader. State: Replica: 3437e5c5400d4f0480e025ab6a174284, State: Running, Role: LEADER
I20260926 02:55:54.835942 1165 consensus_queue.cc:237] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e4d69bb1343dfaf7f50085105d280" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40687 } } peers { permanent_uuid: "c80383aab1b54388b43b9a06b0291f08" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43701 } } peers { permanent_uuid: "3437e5c5400d4f0480e025ab6a174284" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 36985 } }
W20260926 02:55:54.837105 860 tablet.cc:2369] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:55:54.842206 662 catalog_manager.cc:5495] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3437e5c5400d4f0480e025ab6a174284 (127.19.185.194). New cstate: current_term: 1 leader_uuid: "3437e5c5400d4f0480e025ab6a174284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290e4d69bb1343dfaf7f50085105d280" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c80383aab1b54388b43b9a06b0291f08" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3437e5c5400d4f0480e025ab6a174284" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 36985 } health_report { overall_health: HEALTHY } } }
I20260926 02:55:54.895011 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.889591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.19.185.194:36985)
0926 02:55:54.889808 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:54.889815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:54.889835 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:55:54.890107 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:54.890111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:54.890196 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:54.890343 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:55:54.890732 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.891542 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.892926 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.893310 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.893446 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.893923 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:55:54.894439 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:55:54.894602 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:54.894659 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 02:55:54.894737 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:55:54.904249 1091 raft_consensus.cc:1261] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08 [term 1 FOLLOWER]: Refusing update from remote peer 3437e5c5400d4f0480e025ab6a174284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:55:54.904461 814 raft_consensus.cc:1261] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280 [term 1 FOLLOWER]: Refusing update from remote peer 3437e5c5400d4f0480e025ab6a174284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:55:54.905423 1165 consensus_queue.cc:1035] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c80383aab1b54388b43b9a06b0291f08" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:55:54.906064 1170 consensus_queue.cc:1035] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290e4d69bb1343dfaf7f50085105d280" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:55:54.943631 661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:55:54.943991 661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:55:54.948568 661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7eda3c2b2ae44b0bb5ccf46f4c740fc: Sending DeleteTablet for 3 replicas of tablet 77dc61ea73c14a5eb3e4fd1fb5cf092d
I20260926 02:55:54.949949 1072 tablet_service.cc:1514] Processing DeleteTablet for tablet 77dc61ea73c14a5eb3e4fd1fb5cf092d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:55:54 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:44898
I20260926 02:55:54.949940 938 tablet_service.cc:1514] Processing DeleteTablet for tablet 77dc61ea73c14a5eb3e4fd1fb5cf092d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:55:54 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:37886
I20260926 02:55:54.950124 794 tablet_service.cc:1514] Processing DeleteTablet for tablet 77dc61ea73c14a5eb3e4fd1fb5cf092d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:55:54 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:42198
I20260926 02:55:54.956286 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 699
I20260926 02:55:54.972834 1190 tablet_replica.cc:330] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08: stopping tablet replica
I20260926 02:55:54.974014 1190 raft_consensus.cc:2229] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:55:54.975216 1190 raft_consensus.cc:2258] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:55:54.978411 1190 ts_tablet_manager.cc:1905] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:55:54.980145 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:54.957572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.19.185.193:40687)
0926 02:55:54.957748 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:54.957755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:54.957775 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:55:54.957859 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:54.957864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:54.957956 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:54.958069 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:54.963971 (+ 5902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.965194 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.967726 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:54.968434 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:54.968574 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:54.972759 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:54.972776 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:54.972797 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:54.972842 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:54.976851 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:54.977712 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:54.977721 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:54.977725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:54.977921 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:54.978309 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:54.978314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:54.978318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:54.978836 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:54.979156 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:54.979522 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 02:55:54.979760 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:55:54.980782 854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:55:54.986388 1190 ts_tablet_manager.cc:1918] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:55:54.986618 1190 log.cc:1196] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/77dc61ea73c14a5eb3e4fd1fb5cf092d
I20260926 02:55:54.987275 1190 ts_tablet_manager.cc:1939] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P c80383aab1b54388b43b9a06b0291f08: Deleting consensus metadata
I20260926 02:55:54.987746 1189 tablet_replica.cc:330] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284: stopping tablet replica
I20260926 02:55:54.988219 1189 raft_consensus.cc:2229] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:55:54.988919 1189 raft_consensus.cc:2258] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:55:54.989672 647 catalog_manager.cc:4841] TS c80383aab1b54388b43b9a06b0291f08 (127.19.185.195:43701): tablet 77dc61ea73c14a5eb3e4fd1fb5cf092d (table test-table [id=fa98a092780e4096b6ec2e60277a41c6]) successfully deleted
I20260926 02:55:54.991231 1189 ts_tablet_manager.cc:1905] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:55:54.999456 1189 ts_tablet_manager.cc:1918] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:55:54.999678 1189 log.cc:1196] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/77dc61ea73c14a5eb3e4fd1fb5cf092d
I20260926 02:55:55.000217 1189 ts_tablet_manager.cc:1939] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 3437e5c5400d4f0480e025ab6a174284: Deleting consensus metadata
I20260926 02:55:55.002174 646 catalog_manager.cc:4841] TS 3437e5c5400d4f0480e025ab6a174284 (127.19.185.194:36985): tablet 77dc61ea73c14a5eb3e4fd1fb5cf092d (table test-table [id=fa98a092780e4096b6ec2e60277a41c6]) successfully deleted
I20260926 02:55:55.175935 1191 tablet_replica.cc:330] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280: stopping tablet replica
I20260926 02:55:55.176416 1191 raft_consensus.cc:2229] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:55:55.176928 1191 raft_consensus.cc:2258] T 77dc61ea73c14a5eb3e4fd1fb5cf092d P 290e4d69bb1343dfaf7f50085105d280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:55:55.178202 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 699
I20260926 02:55:55.203933 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 863
W20260926 02:55:55.204233 647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.185.193:40687 (error 108)
I20260926 02:55:55.212751 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:55.206585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40687 (local address 127.0.0.1:42214)
0926 02:55:55.212309 (+ 5724us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:55.212470 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5546,"threads_started":1}
W20260926 02:55:55.213924 647 catalog_manager.cc:4584] TS 290e4d69bb1343dfaf7f50085105d280 (127.19.185.193:40687): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77dc61ea73c14a5eb3e4fd1fb5cf092d: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40687: connect: Connection refused (error 111)
I20260926 02:55:55.225793 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:55.205999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.19.185.194:36985)
0926 02:55:55.206233 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:55.206240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:55.206266 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:55:55.206354 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:55.206361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:55.206456 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:55.206578 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:55.208574 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:55.209985 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:55.211935 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:55.212681 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:55.212862 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:55.216945 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:55.216959 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:55.216979 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:55.217033 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:55.221087 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:55.222103 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:55.222112 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:55.222116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:55.222319 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:55.222853 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:55.222860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:55.222863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:55.223363 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:55.224990 (+ 1627us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:55.225178 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 02:55:55.225405 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 02:55:55.226737 998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:55:55.267683 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:55.267117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40687 (local address 127.0.0.1:42226)
0926 02:55:55.267400 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:55.267527 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:55:55.268942 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:55.268483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40687 (local address 127.0.0.1:42240)
0926 02:55:55.268652 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:55.268750 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 02:55:55.329710 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:55.329171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40687 (local address 127.0.0.1:42242)
0926 02:55:55.329392 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:55.329480 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 02:55:55.330837 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:55.330474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40687 (local address 127.0.0.1:42250)
0926 02:55:55.330625 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:55.330704 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 02:55:55.357116 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 863
I20260926 02:55:55.373304 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 1007
I20260926 02:55:55.387063 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:55.374115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.19.185.195:43701)
0926 02:55:55.374641 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:55.374645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:55.374661 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:55:55.374719 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:55.374723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:55.374782 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:55.374926 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:55.376387 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:55.377114 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:55.378312 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:55.378709 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:55.378802 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:55.381364 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:55.381376 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:55.381392 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:55.381425 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:55.384541 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:55.385195 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:55.385202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:55.385204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:55.385323 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:55.385682 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:55.385688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:55.385693 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:55.386074 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:55.386298 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:55.386570 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 02:55:55.386777 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":289,"threads_started":1}
I20260926 02:55:55.388007 1132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:55:55.411278 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:55.410714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40687 (local address 127.0.0.1:42252)
0926 02:55:55.410962 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:55.411093 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 02:55:55.412482 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:55.412028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40687 (local address 127.0.0.1:42266)
0926 02:55:55.412245 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:55.412329 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 02:55:55.478574 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 1007
I20260926 02:55:55.495458 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 629
I20260926 02:55:55.496728 692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:55:55.612246 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:55.611698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40687 (local address 127.0.0.1:42272)
0926 02:55:55.611921 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:55.612035 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 02:55:55.612476 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 629
I20260926 02:55:55.634227 20199 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 02:55:55.969601 20199 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 02:55:55.969780 20199 mini_ranger.cc:66] Stopping Ranger...
I20260926 02:55:56.315253 20199 mini_ranger.cc:68] Stopped Ranger
2026-09-26 02:55:56.315 UTC [32149] LOG: received smart shutdown request
2026-09-26 02:55:56.318 UTC [32149] LOG: background worker "logical replication launcher" (PID 32158) exited with exit code 1
2026-09-26 02:55:56.319 UTC [32153] LOG: shutting down
2026-09-26 02:55:56.499 UTC [32149] LOG: database system is shut down
2026-09-26T02:55:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52878 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:55:56 dist-test-slave-ktng krb5kdc[1207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:55:56 dist-test-slave-ktng krb5kdc[1207](info): set up 2 sockets
Sep 26 02:55:56 dist-test-slave-ktng krb5kdc[1207](info): commencing operation
krb5kdc: starting...
W20260926 02:55:57.761152 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:55:57 dist-test-slave-ktng krb5kdc[1207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:55:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:55:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:55:57.996343 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:40477
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:46195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:40477,127.19.185.253:32809,127.19.185.252:38293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:55:58.327118 1220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:55:58.327564 1220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:55:58.327755 1220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:55:58.335925 1220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:55:58.336097 1220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:55:58.336220 1220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:55:58.336342 1220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:55:58.336448 1220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:55:58.347555 1220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:46195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:40477,127.19.185.253:32809,127.19.185.252:38293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:40477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:55:58.349076 1220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:55:58.350507 1220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:55:58.363593 1225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:58.363708 1226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:58.363729 1228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:58.364032 1220 server_base.cc:1034] running on GCE node
I20260926 02:55:58.365077 1220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:55:58.366513 1220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:55:58.367774 1220 hybrid_clock.cc:648] HybridClock initialized: now 1790391358367716 us; error 45 us; skew 500 ppm
I20260926 02:55:58.368167 1220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:55:58 dist-test-slave-ktng krb5kdc[1207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391358, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:55:58.376698 1220 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:55:58.378317 1220 webserver.cc:460] Webserver started at http://127.19.185.254:44577/ using document root <none> and password file <none>
I20260926 02:55:58.379037 1220 fs_manager.cc:359] Metadata directory not provided
I20260926 02:55:58.379184 1220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:55:58.379562 1220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:55:58.382546 1220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "d6c3da0ea2494b159d5b8b9b83812eb2"
format_stamp: "Formatted at 2026-09-26 02:55:58 on dist-test-slave-ktng"
server_key: "c359631596b9ff7520b70a8be5da878e"
server_key_iv: "59d068e07916b1de524ee7f4c45583ca"
server_key_version: "encryptionkey@0"
I20260926 02:55:58.383404 1220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "d6c3da0ea2494b159d5b8b9b83812eb2"
format_stamp: "Formatted at 2026-09-26 02:55:58 on dist-test-slave-ktng"
server_key: "c359631596b9ff7520b70a8be5da878e"
server_key_iv: "59d068e07916b1de524ee7f4c45583ca"
server_key_version: "encryptionkey@0"
I20260926 02:55:58.388682 1220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:55:58.392004 1235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:58.393429 1220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:55:58.393612 1220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "d6c3da0ea2494b159d5b8b9b83812eb2"
format_stamp: "Formatted at 2026-09-26 02:55:58 on dist-test-slave-ktng"
server_key: "c359631596b9ff7520b70a8be5da878e"
server_key_iv: "59d068e07916b1de524ee7f4c45583ca"
server_key_version: "encryptionkey@0"
I20260926 02:55:58.393807 1220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:55:58.411893 1220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:55:58.416069 1220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:55:58.416345 1220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:55:58.435424 1220 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:40477
I20260926 02:55:58.435525 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:40477 every 8 connection(s)
I20260926 02:55:58.436825 1220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:55:58.439271 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 1220
I20260926 02:55:58.439576 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:55:58.440256 20199 external_mini_cluster.cc:1428] Setting key e973493fbc93d55f0a9d20a1cff0ada4
I20260926 02:55:58.441286 1287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:55:58.448995 1287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.185.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.253@KRBTEST.COM" created.
Sep 26 02:55:58 dist-test-slave-ktng krb5kdc[1207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391358, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:55:58.477263 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.453331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.19.185.254:40477)
0926 02:55:58.455498 (+ 2167us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:58.455505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:58.455545 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:55:58.458215 (+ 2670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:58.458220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:58.458456 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:58.458618 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:58.460581 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:58.461759 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:58.464315 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:58.464955 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:58.465121 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:58.469088 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:58.469101 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:58.469121 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:58.469170 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:58.473045 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:58.473763 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:58.473774 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:58.473780 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:58.473969 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:58.474431 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:58.474434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:58.474436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:58.474917 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:58.475078 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:58.475861 (+ 783us) server_negotiation.cc:299] Negotiation successful
0926 02:55:58.476185 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1971,"spinlock_wait_cycles":74112,"thread_start_us":1621,"threads_started":1}
I20260926 02:55:58.477306 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.453331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:40477 (local address 127.0.0.1:33348)
0926 02:55:58.454373 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.454431 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:58.457768 (+ 3337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:58.458742 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:58.458753 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:58.459727 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:58.460350 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:58.460386 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:58.461949 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:58.461956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:58.464173 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:58.464195 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:58.465205 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:58.465210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:58.465354 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:58.466137 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:58.466162 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:58.468825 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:58.473175 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:58.473185 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:58.473195 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:58.473614 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:58.474098 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:58.474102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:58.474104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:58.474303 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:58.475067 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:58.475078 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:58.475427 (+ 349us) client_negotiation.cc:769] Sending connection context
0926 02:55:58.475742 (+ 315us) client_negotiation.cc:240] Negotiation successful
0926 02:55:58.476394 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":82,"thread_start_us":430,"threads_started":1}
I20260926 02:55:58.480453 1287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:55:58.481665 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.481116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:32809 (local address 127.0.0.1:56824)
0926 02:55:58.481330 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.481487 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 02:55:58.481918 1239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
I20260926 02:55:58.483698 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.483184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:32809 (local address 127.0.0.1:56830)
0926 02:55:58.483428 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.483527 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 02:55:58.483954 1287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:32809: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.253.keytab.
Entry for principal kudu/127.19.185.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.253.keytab.
I20260926 02:55:58.499693 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.253:32809
--webserver_interface=127.19.185.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.253.keytab
--principal=kudu/127.19.185.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:46195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:40477,127.19.185.253:32809,127.19.185.252:38293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:55:58.533362 1287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } attempt: 1
I20260926 02:55:58.534693 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.534124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:32809 (local address 127.0.0.1:56834)
0926 02:55:58.534364 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.534513 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 02:55:58.535889 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.535447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:32809 (local address 127.0.0.1:56844)
0926 02:55:58.535605 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.535719 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 02:55:58.536118 1287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:32809: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
I20260926 02:55:58.604389 1287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } attempt: 2
I20260926 02:55:58.605677 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.605104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:32809 (local address 127.0.0.1:56854)
0926 02:55:58.605377 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.605500 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 02:55:58.606803 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.606329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:32809 (local address 127.0.0.1:56870)
0926 02:55:58.606509 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.606609 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 02:55:58.607013 1287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:32809: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
I20260926 02:55:58.698274 1287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } attempt: 3
I20260926 02:55:58.699559 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.698998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:32809 (local address 127.0.0.1:56874)
0926 02:55:58.699227 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.699388 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 02:55:58.700666 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.700265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:32809 (local address 127.0.0.1:56890)
0926 02:55:58.700431 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.700516 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:55:58.700940 1287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:32809: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
W20260926 02:55:58.831876 1294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:55:58.832244 1294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:55:58.832334 1294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:55:58.839434 1294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:55:58.839537 1294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:55:58.839583 1294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:55:58.839632 1294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:55:58.839679 1294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:55:58.844233 1287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } attempt: 4
I20260926 02:55:58.845461 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.844925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:32809 (local address 127.0.0.1:56904)
0926 02:55:58.845238 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.845324 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260926 02:55:58.846531 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.846165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:32809 (local address 127.0.0.1:56916)
0926 02:55:58.846319 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.846407 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:55:58.846763 1287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:32809: Network error: Client connection negotiation failed: client connection to 127.19.185.253:32809: connect: Connection refused (error 111)
I20260926 02:55:58.850543 1294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:46195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:40477,127.19.185.253:32809,127.19.185.252:38293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.253.keytab
--principal=kudu/127.19.185.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.253:32809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.185.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:55:58.851938 1294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:55:58.853195 1294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:55:58.864852 1299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:58.864877 1300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:58.865077 1302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:58.865921 1294 server_base.cc:1034] running on GCE node
I20260926 02:55:58.866675 1294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:55:58.868055 1294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:55:58.869285 1294 hybrid_clock.cc:648] HybridClock initialized: now 1790391358869246 us; error 49 us; skew 500 ppm
I20260926 02:55:58.869622 1294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:55:58 dist-test-slave-ktng krb5kdc[1207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391358, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:55:58.872705 1294 init.cc:377] Logged in from keytab as kudu/127.19.185.253@KRBTEST.COM (short username kudu)
I20260926 02:55:58.874152 1294 webserver.cc:460] Webserver started at http://127.19.185.253:35769/ using document root <none> and password file <none>
I20260926 02:55:58.874878 1294 fs_manager.cc:359] Metadata directory not provided
I20260926 02:55:58.875023 1294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:55:58.875367 1294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:55:58.878310 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/data/instance:
uuid: "b3b6ae4086f845dab41100f6e93c61c9"
format_stamp: "Formatted at 2026-09-26 02:55:58 on dist-test-slave-ktng"
server_key: "7d24d05a9f7889cd1378b7cdb8fe5a9b"
server_key_iv: "654a5a127e84d49f5449946e9a930b11"
server_key_version: "encryptionkey@0"
I20260926 02:55:58.879120 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/wal/instance:
uuid: "b3b6ae4086f845dab41100f6e93c61c9"
format_stamp: "Formatted at 2026-09-26 02:55:58 on dist-test-slave-ktng"
server_key: "7d24d05a9f7889cd1378b7cdb8fe5a9b"
server_key_iv: "654a5a127e84d49f5449946e9a930b11"
server_key_version: "encryptionkey@0"
I20260926 02:55:58.884395 1294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:55:58.887936 1309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:58.889252 1294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 02:55:58.889436 1294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/wal
uuid: "b3b6ae4086f845dab41100f6e93c61c9"
format_stamp: "Formatted at 2026-09-26 02:55:58 on dist-test-slave-ktng"
server_key: "7d24d05a9f7889cd1378b7cdb8fe5a9b"
server_key_iv: "654a5a127e84d49f5449946e9a930b11"
server_key_version: "encryptionkey@0"
I20260926 02:55:58.889635 1294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:55:58.915755 1294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:55:58.920473 1294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:55:58.920830 1294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:55:58.940574 1294 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.253:32809
I20260926 02:55:58.940605 1360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.253:32809 every 8 connection(s)
I20260926 02:55:58.942113 1294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/data/info.pb
I20260926 02:55:58.945747 1361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:55:58.952390 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 1294
I20260926 02:55:58.952601 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-1/wal/instance
I20260926 02:55:58.952060 1361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:55:58.953109 20199 external_mini_cluster.cc:1428] Setting key 570efa70b552a3e739529de792d470b1
Sep 26 02:55:58 dist-test-slave-ktng krb5kdc[1207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391358, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.253@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:55:58.971597 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.954765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.19.185.254:40477)
0926 02:55:58.954985 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:58.954999 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:58.956268 (+ 1269us) server_negotiation.cc:407] Connection header received
0926 02:55:58.958129 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:58.958138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:58.958232 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:58.958351 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:58.959877 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:58.960554 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:58.962065 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:58.962478 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:58.962621 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:58.965950 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:58.965955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:58.965958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:58.965992 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:58.968610 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:58.969330 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:58.969340 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:58.969343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:58.969458 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:58.969922 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:58.969926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:58.969928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:58.970196 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:58.970348 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:58.971009 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 02:55:58.971233 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.19.185.252@KRBTEST.COM; defaulting to no policy
I20260926 02:55:58.972577 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.955233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:40477 (local address 127.0.0.1:33356)
0926 02:55:58.956085 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.956131 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:58.957851 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:58.958550 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:58.958557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:58.959081 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:58.959697 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:58.959716 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:58.960720 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:58.960724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:58.961900 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:58.961912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:58.962719 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:58.962726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:58.962948 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:58.963661 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:58.963687 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:58.965725 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:58.968752 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:58.968760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:58.968776 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:58.969202 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:58.969593 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:58.969597 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:58.969601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:58.969799 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:58.970319 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:58.970327 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:58.970650 (+ 323us) client_negotiation.cc:769] Sending connection context
0926 02:55:58.970909 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 02:55:58.971426 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":341,"threads_started":1}
I20260926 02:55:58.974308 1361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.252@KRBTEST.COM" created.
Sep 26 02:55:58 dist-test-slave-ktng krb5kdc[1207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391358, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.253@KRBTEST.COM for kudu/127.19.185.253@KRBTEST.COM
I20260926 02:55:58.989691 1365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.975284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56918 (local address 127.19.185.253:32809)
0926 02:55:58.975909 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:58.975914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:58.975938 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:55:58.975977 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:58.975983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:58.976158 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:58.976307 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:58.977002 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:58.978048 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:58.979217 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:58.979831 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:58.979967 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:58.983214 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:58.983228 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:58.983250 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:58.983317 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:58.986389 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:58.987425 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:58.987433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:58.987436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:58.987562 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:58.988007 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:58.988010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:58.988013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:58.988412 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:58.988573 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:58.989092 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 02:55:58.989376 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":463,"thread_start_us":323,"threads_started":1}
I20260926 02:55:58.989784 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.974978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:32809 (local address 127.0.0.1:56918)
0926 02:55:58.975196 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.975210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:58.975285 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:58.976299 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:58.976303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:58.976726 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:58.976850 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:58.976858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:58.978180 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:58.978188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:58.979088 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:58.979099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:58.979974 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:58.979979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:58.980240 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:58.980837 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:58.980853 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:58.982969 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:58.986554 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:58.986562 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:58.986569 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:58.987132 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:58.987686 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:58.987694 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:58.987701 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:58.987886 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:58.988568 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:58.988576 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:58.988750 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 02:55:58.988907 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 02:55:58.989254 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":24}
I20260926 02:55:58.992666 1361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:55:58.994016 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.993416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58486)
0926 02:55:58.993660 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.993807 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 02:55:58.994297 1311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
I20260926 02:55:58.996125 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:58.995657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58488)
0926 02:55:58.995850 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:58.995971 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 02:55:58.996409 1361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:38293: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.252.keytab.
Entry for principal kudu/127.19.185.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.252.keytab.
I20260926 02:55:59.007709 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.252:38293
--webserver_interface=127.19.185.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.252.keytab
--principal=kudu/127.19.185.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:46195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:40477,127.19.185.253:32809,127.19.185.252:38293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:55:59.045769 1361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } attempt: 1
I20260926 02:55:59.047075 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.046513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58504)
0926 02:55:59.046759 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.046889 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:55:59.048202 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.047790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58516)
0926 02:55:59.047981 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.048073 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260926 02:55:59.048420 1361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:38293: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
I20260926 02:55:59.116667 1361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } attempt: 2
I20260926 02:55:59.118019 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.117368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58518)
0926 02:55:59.117639 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.117788 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 02:55:59.119334 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.118786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58532)
0926 02:55:59.118980 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.119095 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 02:55:59.119583 1361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:38293: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
I20260926 02:55:59.146036 1287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } attempt: 5
Sep 26 02:55:59 dist-test-slave-ktng krb5kdc[1207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391358, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for kudu/127.19.185.253@KRBTEST.COM
I20260926 02:55:59.166340 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.146852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:32809 (local address 127.0.0.1:56922)
0926 02:55:59.147080 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.147104 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:59.147242 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:59.147688 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:59.147691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:59.148051 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:59.148222 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:59.148230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.149639 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.149642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:59.150512 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:59.150523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.151682 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.151686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:59.151818 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.156762 (+ 4944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:59.156781 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:59.158899 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:59.163149 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:59.163159 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:59.163164 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:59.163804 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:59.164270 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:59.164276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:59.164283 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:59.164498 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:59.165351 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:59.165356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:59.165533 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 02:55:59.165681 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 02:55:59.165975 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 02:55:59.168249 1365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.147047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.19.185.253:32809)
0926 02:55:59.147253 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:59.147259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:59.147306 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:55:59.147452 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:59.147458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:59.147540 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:59.147654 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:59.148395 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.149497 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.150661 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.151539 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.151679 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.159114 (+ 7435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:59.159122 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:59.159126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:59.159168 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:59.162948 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.163957 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.163963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.163967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.164136 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.164721 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.164726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.164729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.165103 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:59.165297 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:59.166815 (+ 1518us) server_negotiation.cc:299] Negotiation successful
0926 02:55:59.167042 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:55:59.169457 1287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:55:59.171216 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.170624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58538)
0926 02:55:59.170920 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.171029 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260926 02:55:59.172432 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.172001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58552)
0926 02:55:59.172215 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.172304 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260926 02:55:59.172689 1287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:38293: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
I20260926 02:55:59.210803 1361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } attempt: 3
I20260926 02:55:59.211966 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.211530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58554)
0926 02:55:59.211709 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.211806 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 02:55:59.213187 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.212784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58564)
0926 02:55:59.213005 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.213076 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260926 02:55:59.213459 1361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:38293: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
I20260926 02:55:59.223968 1287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } attempt: 1
I20260926 02:55:59.225159 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.224589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58568)
0926 02:55:59.224825 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.224960 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 02:55:59.226248 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.225836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58576)
0926 02:55:59.226010 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.226104 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 02:55:59.226472 1287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:38293: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
I20260926 02:55:59.294682 1287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } attempt: 2
I20260926 02:55:59.295996 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.295437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58584)
0926 02:55:59.295674 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.295797 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 02:55:59.297116 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.296697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58588)
0926 02:55:59.296839 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.296969 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 02:55:59.297336 1287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:38293: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
I20260926 02:55:59.356691 1361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } attempt: 4
I20260926 02:55:59.357864 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.357338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58592)
0926 02:55:59.357567 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.357672 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 02:55:59.358840 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.358524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58600)
0926 02:55:59.358667 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.358742 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 02:55:59.359083 1361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:38293: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
W20260926 02:55:59.362103 1368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:55:59.362416 1368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:55:59.362576 1368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:55:59.369822 1368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:55:59.369948 1368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:55:59.370048 1368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:55:59.370142 1368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:55:59.370203 1368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:55:59.380025 1368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:46195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:40477,127.19.185.253:32809,127.19.185.252:38293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.252.keytab
--principal=kudu/127.19.185.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.252:38293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.185.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:55:59.381309 1368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:55:59.382566 1368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:55:59.394440 1374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:59.394549 1373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:55:59.394650 1376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:55:59.395438 1368 server_base.cc:1034] running on GCE node
I20260926 02:55:59.396167 1368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:55:59.397696 1368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:55:59.398984 1368 hybrid_clock.cc:648] HybridClock initialized: now 1790391359398910 us; error 76 us; skew 500 ppm
I20260926 02:55:59.399504 1368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:55:59 dist-test-slave-ktng krb5kdc[1207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391359, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:55:59.402837 1368 init.cc:377] Logged in from keytab as kudu/127.19.185.252@KRBTEST.COM (short username kudu)
I20260926 02:55:59.403551 1287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } attempt: 3
I20260926 02:55:59.404450 1368 webserver.cc:460] Webserver started at http://127.19.185.252:33575/ using document root <none> and password file <none>
I20260926 02:55:59.404536 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.404092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58604)
0926 02:55:59.404307 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.404383 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 02:55:59.405370 1368 fs_manager.cc:359] Metadata directory not provided
I20260926 02:55:59.405531 1368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:55:59.405601 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.405199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58620)
0926 02:55:59.405387 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.405456 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260926 02:55:59.405818 1287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:38293: Network error: Client connection negotiation failed: client connection to 127.19.185.252:38293: connect: Connection refused (error 111)
I20260926 02:55:59.405891 1368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:55:59.408825 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/data/instance:
uuid: "d288357a2c854f6282ee082f92ab0015"
format_stamp: "Formatted at 2026-09-26 02:55:59 on dist-test-slave-ktng"
server_key: "7677cbee9f01c7eb23e1a5ae018e1815"
server_key_iv: "5a07576285205109aed522b388bbb73f"
server_key_version: "encryptionkey@0"
I20260926 02:55:59.409618 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/wal/instance:
uuid: "d288357a2c854f6282ee082f92ab0015"
format_stamp: "Formatted at 2026-09-26 02:55:59 on dist-test-slave-ktng"
server_key: "7677cbee9f01c7eb23e1a5ae018e1815"
server_key_iv: "5a07576285205109aed522b388bbb73f"
server_key_version: "encryptionkey@0"
I20260926 02:55:59.414597 1368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:55:59.418064 1383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:55:59.419427 1368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:55:59.419612 1368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/wal
uuid: "d288357a2c854f6282ee082f92ab0015"
format_stamp: "Formatted at 2026-09-26 02:55:59 on dist-test-slave-ktng"
server_key: "7677cbee9f01c7eb23e1a5ae018e1815"
server_key_iv: "5a07576285205109aed522b388bbb73f"
server_key_version: "encryptionkey@0"
I20260926 02:55:59.419798 1368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:55:59.433331 1368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:55:59.438446 1368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:55:59.438750 1368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:55:59.458302 1368 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.252:38293
I20260926 02:55:59.458324 1434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.252:38293 every 8 connection(s)
I20260926 02:55:59.460260 1368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/data/info.pb
I20260926 02:55:59.463923 1435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:55:59.469892 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 1368
I20260926 02:55:59.470098 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391151502216-20199-0/minicluster-data/master-2/wal/instance
I20260926 02:55:59.470610 20199 external_mini_cluster.cc:1428] Setting key 5c5de1c4b52bedc109cb8f842ba4323f
I20260926 02:55:59.470213 1435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:55:59 dist-test-slave-ktng krb5kdc[1207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391359, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.252@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
Sep 26 02:55:59 dist-test-slave-ktng krb5kdc[1207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.252@KRBTEST.COM
I20260926 02:55:59.492044 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.472898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.19.185.254:40477)
0926 02:55:59.473423 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:59.473429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:59.473890 (+ 461us) server_negotiation.cc:407] Connection header received
0926 02:55:59.475975 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:59.475981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:59.476075 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:59.476209 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:59.477456 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.478408 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.480158 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.480967 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.481075 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.484601 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:59.484610 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:59.484615 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:59.484665 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:59.488393 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.489249 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.489254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.489258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.489419 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.489852 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.489860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.489864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.490251 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:59.490418 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:59.491344 (+ 926us) server_negotiation.cc:299] Negotiation successful
0926 02:55:59.491593 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":238,"threads_started":1}
I20260926 02:55:59.492830 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.472932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:40477 (local address 127.0.0.1:33366)
0926 02:55:59.473729 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.473777 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:59.475639 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:59.476278 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:59.476285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:59.476918 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:59.477277 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:59.477294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.478581 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.478587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:59.480003 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:59.480017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.481181 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.481188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:59.481524 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.482337 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:59.482357 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:59.484398 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:59.488579 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:59.488588 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:59.488597 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:59.489104 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:59.489536 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:59.489539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:59.489541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:59.489714 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:59.490389 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:59.490396 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:59.490842 (+ 446us) client_negotiation.cc:769] Sending connection context
0926 02:55:59.491149 (+ 307us) client_negotiation.cc:240] Negotiation successful
0926 02:55:59.491663 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":50,"thread_start_us":443,"threads_started":1}
I20260926 02:55:59.493721 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.472574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.19.185.252:38293)
0926 02:55:59.473933 (+ 1359us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:59.473940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:59.473982 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:55:59.475629 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:59.475649 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:59.475868 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:59.476030 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:59.477674 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.479119 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.480909 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.481729 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.481872 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.486460 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:59.486469 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:59.486506 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:59.486563 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:59.490224 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.491099 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.491106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.491108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.491349 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.491781 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.491787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.491791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.492268 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:59.492596 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:59.492991 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 02:55:59.493342 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":1117,"thread_start_us":1200,"threads_started":1}
I20260926 02:55:59.494912 1435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:55:59 dist-test-slave-ktng krb5kdc[1207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391359, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.252@KRBTEST.COM for kudu/127.19.185.253@KRBTEST.COM
Sep 26 02:55:59 dist-test-slave-ktng krb5kdc[1207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:55:59.508762 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.495668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:32809 (local address 127.0.0.1:56928)
0926 02:55:59.495841 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.495862 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:59.495957 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:59.496312 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:59.496315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:59.496685 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:59.496888 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:59.496899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.497941 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.497949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:59.498917 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:59.498929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.499732 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.499737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:59.499960 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.500542 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:59.500558 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:59.503068 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:59.506116 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:59.506125 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:59.506132 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:59.506554 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:59.506994 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:59.507003 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:59.507008 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:59.507229 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:59.507886 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:59.507891 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:59.508067 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 02:55:59.508242 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 02:55:59.508508 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 02:55:59.508970 1365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.495797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.19.185.253:32809)
0926 02:55:59.495990 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:59.495996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:59.496018 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:55:59.496087 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:59.496094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:59.496176 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:59.496286 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:59.497029 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.497786 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.499066 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.499586 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.499723 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.503256 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:59.503261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:59.503264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:59.503341 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:59.505971 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.506690 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.506696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.506699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.506870 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.507374 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.507380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.507384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.507746 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:59.507903 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:59.508355 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 02:55:59.508584 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:55:59.509867 1435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:55:59.512822 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.496733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33376 (local address 127.19.185.254:40477)
0926 02:55:59.496912 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:59.496917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:59.496936 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:55:59.497111 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:59.497117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:59.497238 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:59.497353 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:59.498808 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.499662 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.501147 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.501670 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.501831 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.506222 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:59.506226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:59.506229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:59.506259 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:59.509813 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.510675 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.510682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.510686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.510824 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.511239 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.511250 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.511252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.511597 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:59.511940 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:59.512238 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 02:55:59.512510 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 02:55:59 dist-test-slave-ktng krb5kdc[1207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391359, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.252@KRBTEST.COM for kudu/127.19.185.252@KRBTEST.COM
I20260926 02:55:59.524201 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.510843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58640)
0926 02:55:59.511063 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.511077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:59.511195 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:59.511540 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:59.511545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:59.511907 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:59.512036 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:59.512043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.513136 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.513141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:59.514383 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:59.514395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.515152 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.515156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:59.515468 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.516037 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:59.516061 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:59.518432 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:59.521563 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:59.521569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:59.521576 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:59.522084 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:59.522463 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:59.522469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:59.522474 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:59.522650 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:59.523123 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:59.523129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:59.523334 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 02:55:59.523514 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 02:55:59.523775 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":3}
Sep 26 02:55:59 dist-test-slave-ktng krb5kdc[1207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.253@KRBTEST.COM
I20260926 02:55:59.524315 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.510843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58640 (local address 127.19.185.252:38293)
0926 02:55:59.511019 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:59.511024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:59.511140 (+ 116us) server_negotiation.cc:407] Connection header received
0926 02:55:59.511348 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:59.511353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:59.511422 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:59.511517 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:59.512162 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.512966 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.514525 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.515011 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.515150 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.518611 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:59.518616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:59.518619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:59.518646 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:59.521422 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.522196 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.522200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.522202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.522326 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.522760 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.522763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.522765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.523032 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:59.523165 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:59.523674 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 02:55:59.523946 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":63}
I20260926 02:55:59.529433 1435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015: Bootstrap starting.
I20260926 02:55:59.531019 1365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.514873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.19.185.253:32809)
0926 02:55:59.515075 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:59.515084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:59.515105 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:55:59.515159 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:59.515164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:59.515280 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:59.515437 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:59.517046 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.517843 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.519356 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.520047 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.520197 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.525509 (+ 5312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:59.525515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:59.525519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:59.525558 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:59.528179 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.528974 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.528977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.528979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.529137 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.529628 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.529634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.529638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.529975 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:59.530205 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:59.530570 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 02:55:59.530742 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:55:59.533711 1435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015: Neither blocks nor log segments found. Creating new log.
I20260926 02:55:59.534896 1435 log.cc:824] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015: Log is configured to *not* fsync() on all Append() calls
I20260926 02:55:59.536211 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:55:59.537745 1435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015: No bootstrap required, opened a new log
I20260926 02:55:59.548153 1435 raft_consensus.cc:348] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } }
I20260926 02:55:59.548710 1435 raft_consensus.cc:374] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:55:59.548899 1435 raft_consensus.cc:729] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d288357a2c854f6282ee082f92ab0015, State: Initialized, Role: FOLLOWER
I20260926 02:55:59.549762 1435 consensus_queue.cc:260] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } }
I20260926 02:55:59.560999 1435 sys_catalog.cc:564] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:55:59.561658 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.540993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33388 (local address 127.19.185.254:40477)
0926 02:55:59.541188 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:59.541194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:59.541763 (+ 569us) server_negotiation.cc:407] Connection header received
0926 02:55:59.542058 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:59.542065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:59.542150 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:59.542222 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:59.544519 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.545626 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.547806 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.548448 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.548565 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.553202 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:59.553212 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:59.553220 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:59.553275 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:59.557629 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.559261 (+ 1632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.559266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.559269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.559451 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.559894 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.559899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.559902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.560254 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:59.560506 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:59.561047 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 02:55:59.561280 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:55:59.562296 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.541417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.19.185.252:38293)
0926 02:55:59.541594 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:59.541602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:59.542070 (+ 468us) server_negotiation.cc:407] Connection header received
0926 02:55:59.542325 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:59.542332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:59.542414 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:59.542504 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:59.545128 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.546607 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.549429 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.550199 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.550301 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.553915 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:59.553921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:59.553926 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:59.553963 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:59.558235 (+ 4272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.559715 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.559723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.559729 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.559898 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.560363 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.560370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.560375 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.560842 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:59.561074 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:59.561666 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 02:55:59.561890 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:55:59.565331 1449 sys_catalog.cc:455] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } } }
I20260926 02:55:59.565716 1449 sys_catalog.cc:458] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:55:59.567408 1365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.542168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56948 (local address 127.19.185.253:32809)
0926 02:55:59.542333 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:59.542339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:59.549139 (+ 6800us) server_negotiation.cc:407] Connection header received
0926 02:55:59.549522 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:59.549529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:59.549604 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:59.549666 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:59.552007 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.553166 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.555112 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.555841 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.555942 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.559573 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:59.559579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:59.559584 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:59.559627 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:59.563103 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.564954 (+ 1851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.564961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.564966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.565209 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.565819 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.565824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.565829 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.566221 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:59.566477 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:59.566786 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 02:55:59.567030 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 02:55:59.574043 1460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:55:59.574175 1460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:55:59.572242 1368 master_runner.cc:186] Error getting master registration for 127.19.185.254:40477: OK, instance_id { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" instance_seqno: 1790391358429780 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:55:59.580883 1368 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:55:59.583002 1287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } attempt: 4
Sep 26 02:55:59 dist-test-slave-ktng krb5kdc[1207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391358, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for kudu/127.19.185.252@KRBTEST.COM
I20260926 02:55:59.598114 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.583558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58660)
0926 02:55:59.583737 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.583758 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:59.583872 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:59.584293 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:59.584299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:59.584807 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:59.585037 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:59.585050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.586572 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.586578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:59.587711 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:59.587725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.588535 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.588541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:59.588716 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.589326 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:59.589342 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:59.591332 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:59.594599 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:59.594604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:59.594608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:59.594990 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:59.595492 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:59.595497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:59.595500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:59.595663 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:59.597179 (+ 1516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:59.597183 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:59.597378 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 02:55:59.597495 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 02:55:59.597758 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 02:55:59.598232 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.583637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.19.185.252:38293)
0926 02:55:59.583794 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:59.583798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:59.583814 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:55:59.584078 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:59.584082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:59.584155 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:59.584231 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:59.585189 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.586380 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.587868 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.588381 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.588517 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.591519 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:59.591524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:59.591527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:59.591556 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:59.594469 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.595116 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.595128 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.595131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.595282 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.596590 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.596596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.596600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.597056 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:59.597213 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:59.597613 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 02:55:59.597873 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:55:59.602731 1287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2: Bootstrap starting.
I20260926 02:55:59.607772 1287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2: Neither blocks nor log segments found. Creating new log.
I20260926 02:55:59.609134 1287 log.cc:824] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2: Log is configured to *not* fsync() on all Append() calls
I20260926 02:55:59.612411 1287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2: No bootstrap required, opened a new log
I20260926 02:55:59.623677 1287 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } }
I20260926 02:55:59.624089 1287 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:55:59.624208 1287 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c3da0ea2494b159d5b8b9b83812eb2, State: Initialized, Role: FOLLOWER
I20260926 02:55:59.624866 1287 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } }
I20260926 02:55:59.625798 1463 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } } }
I20260926 02:55:59.626178 1463 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:55:59.627216 1287 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 02:55:59.639128 1474 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:55:59.639429 1474 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:55:59.636650 1220 master_runner.cc:186] Error getting master registration for 127.19.185.253:32809: OK, instance_id { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" instance_seqno: 1790391358934791 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:55:59.645562 1220 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:55:59.658329 1361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } attempt: 5
Sep 26 02:55:59 dist-test-slave-ktng krb5kdc[1207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391358, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.253@KRBTEST.COM for kudu/127.19.185.252@KRBTEST.COM
W20260926 02:55:59.670333 1241 tablet.cc:2369] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:55:59.676594 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.658962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:38293 (local address 127.0.0.1:58676)
0926 02:55:59.659248 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 02:55:59.659270 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:55:59.659401 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:55:59.659784 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:55:59.659789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:55:59.660338 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:55:59.660576 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:59.660589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.662366 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.662372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:59.663822 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:55:59.663838 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.664866 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.664872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:55:59.665112 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.665811 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:55:59.665835 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:55:59.669566 (+ 3731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:55:59.673709 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:59.673715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:59.673719 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:59.674217 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:59.674640 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:55:59.674646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:55:59.674650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:55:59.674845 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:55:59.675566 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:55:59.675571 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:55:59.675741 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 02:55:59.675901 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 02:55:59.676200 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260926 02:55:59.676716 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:55:59.659082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.19.185.252:38293)
0926 02:55:59.659315 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 02:55:59.659321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:55:59.659345 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:55:59.659543 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:55:59.659549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:55:59.659628 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:55:59.659742 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:55:59.660776 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.662176 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.663998 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:55:59.664701 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:55:59.664835 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:55:59.669808 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:55:59.669816 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:55:59.669821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:55:59.669868 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:55:59.673570 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.674366 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.674374 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.674382 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.674519 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:55:59.674971 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:55:59.674977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:55:59.674980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:55:59.675403 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:55:59.675564 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:55:59.676033 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 02:55:59.676306 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 02:55:59.683068 1361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9: Bootstrap starting.
I20260926 02:55:59.688090 1361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9: Neither blocks nor log segments found. Creating new log.
I20260926 02:55:59.689531 1361 log.cc:824] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9: Log is configured to *not* fsync() on all Append() calls
W20260926 02:55:59.691048 1389 tablet.cc:2369] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:55:59.692970 1361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9: No bootstrap required, opened a new log
I20260926 02:55:59.705116 1361 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } }
I20260926 02:55:59.705539 1361 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:55:59.705662 1361 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b6ae4086f845dab41100f6e93c61c9, State: Initialized, Role: FOLLOWER
I20260926 02:55:59.706384 1361 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } }
I20260926 02:55:59.707404 1477 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } } }
I20260926 02:55:59.707847 1477 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:55:59.708729 1361 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:55:59.724678 1294 master_runner.cc:263] No leader master found from master b3b6ae4086f845dab41100f6e93c61c9
I20260926 02:55:59.724833 1294 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:55:59.726922 1488 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:55:59.727165 1488 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:55:59.814018 1449 raft_consensus.cc:482] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:55:59.814358 1449 raft_consensus.cc:504] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } }
I20260926 02:55:59.816367 1449 leader_election.cc:288] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6c3da0ea2494b159d5b8b9b83812eb2 (127.19.185.254:40477), b3b6ae4086f845dab41100f6e93c61c9 (127.19.185.253:32809)
I20260926 02:55:59.817651 1262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d288357a2c854f6282ee082f92ab0015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" is_pre_election: true
I20260926 02:55:59.818303 1262 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d288357a2c854f6282ee082f92ab0015 in term 0.
I20260926 02:55:59.819288 1336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d288357a2c854f6282ee082f92ab0015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b6ae4086f845dab41100f6e93c61c9" is_pre_election: true
I20260926 02:55:59.819859 1336 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d288357a2c854f6282ee082f92ab0015 in term 0.
I20260926 02:55:59.820816 1387 leader_election.cc:302] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b6ae4086f845dab41100f6e93c61c9, d288357a2c854f6282ee082f92ab0015; no voters:
I20260926 02:55:59.821663 1449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:55:59.821807 1449 raft_consensus.cc:482] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:55:59.821918 1449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:55:59.830247 1449 raft_consensus.cc:504] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } }
I20260926 02:55:59.833289 1262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d288357a2c854f6282ee082f92ab0015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2"
I20260926 02:55:59.833531 1262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:55:59.837113 1262 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d288357a2c854f6282ee082f92ab0015 in term 1.
I20260926 02:55:59.837642 1387 leader_election.cc:302] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d288357a2c854f6282ee082f92ab0015, d6c3da0ea2494b159d5b8b9b83812eb2; no voters:
I20260926 02:55:59.840631 1449 leader_election.cc:288] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [CANDIDATE]: Term 1 election: Requested vote from peers d6c3da0ea2494b159d5b8b9b83812eb2 (127.19.185.254:40477), b3b6ae4086f845dab41100f6e93c61c9 (127.19.185.253:32809)
I20260926 02:55:59.841168 1449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:55:59.841572 1449 raft_consensus.cc:686] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [term 1 LEADER]: Becoming Leader. State: Replica: d288357a2c854f6282ee082f92ab0015, State: Running, Role: LEADER
I20260926 02:55:59.842371 1449 consensus_queue.cc:237] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } }
I20260926 02:55:59.849465 1336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d288357a2c854f6282ee082f92ab0015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b6ae4086f845dab41100f6e93c61c9"
I20260926 02:55:59.849749 1336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:55:59.853353 1336 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d288357a2c854f6282ee082f92ab0015 in term 1.
I20260926 02:55:59.861421 1490 sys_catalog.cc:455] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d288357a2c854f6282ee082f92ab0015. Latest consensus state: current_term: 1 leader_uuid: "d288357a2c854f6282ee082f92ab0015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } } }
I20260926 02:55:59.862082 1490 sys_catalog.cc:458] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [sys.catalog]: This master's current role is: LEADER
I20260926 02:55:59.867410 1492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:55:59.869737 1492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:55:59.889746 1336 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [term 1 FOLLOWER]: Refusing update from remote peer d288357a2c854f6282ee082f92ab0015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:55:59.890901 1262 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [term 1 FOLLOWER]: Refusing update from remote peer d288357a2c854f6282ee082f92ab0015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:55:59.891978 1449 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:55:59.894594 1449 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 02:55:59.923728 1315 tablet.cc:2369] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:55:59.955565 1463 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d288357a2c854f6282ee082f92ab0015. Latest consensus state: current_term: 1 leader_uuid: "d288357a2c854f6282ee082f92ab0015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } } }
I20260926 02:55:59.955925 1463 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:55:59.964780 1495 mvcc.cc:205] Tried to move back new op lower bound from 7333443010093387776 to 7333443009926291456. Current Snapshot: MvccSnapshot[applied={T|T < 7333443010093387776}]
I20260926 02:55:59.972702 1477 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d288357a2c854f6282ee082f92ab0015. Latest consensus state: current_term: 1 leader_uuid: "d288357a2c854f6282ee082f92ab0015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } } }
I20260926 02:55:59.973035 1477 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:55:59.978857 1490 sys_catalog.cc:455] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d288357a2c854f6282ee082f92ab0015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } } }
I20260926 02:55:59.979146 1490 sys_catalog.cc:458] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [sys.catalog]: This master's current role is: LEADER
I20260926 02:55:59.994424 1449 sys_catalog.cc:455] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d288357a2c854f6282ee082f92ab0015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } } }
I20260926 02:55:59.994992 1449 sys_catalog.cc:458] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015 [sys.catalog]: This master's current role is: LEADER
I20260926 02:56:00.000933 1494 mvcc.cc:205] Tried to move back new op lower bound from 7333443010093387776 to 7333443009926291456. Current Snapshot: MvccSnapshot[applied={T|T < 7333443010093387776}]
I20260926 02:56:00.012540 1492 catalog_manager.cc:1347] Generated new cluster ID: bbfbb64d808e42ef8cd83eed0dedb959
I20260926 02:56:00.012698 1492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:56:00.030678 1463 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d288357a2c854f6282ee082f92ab0015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } } }
I20260926 02:56:00.030956 1463 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:56:00.036432 1477 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d288357a2c854f6282ee082f92ab0015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3da0ea2494b159d5b8b9b83812eb2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40477 } } peers { permanent_uuid: "b3b6ae4086f845dab41100f6e93c61c9" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 32809 } } peers { permanent_uuid: "d288357a2c854f6282ee082f92ab0015" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 38293 } } }
I20260926 02:56:00.036696 1477 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:56:00.067909 1492 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:56:00.070819 1492 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:56:00.100806 1492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d288357a2c854f6282ee082f92ab0015: Generated new TSK 0
I20260926 02:56:00.102104 1492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:56:00.641379 1474 catalog_manager.cc:1259] Loaded cluster ID: bbfbb64d808e42ef8cd83eed0dedb959
I20260926 02:56:00.641474 1474 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2: loading cluster ID for follower catalog manager: success
I20260926 02:56:00.643925 1474 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2: acquiring CA information for follower catalog manager: success
I20260926 02:56:00.645687 1474 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d6c3da0ea2494b159d5b8b9b83812eb2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:56:00.729622 1488 catalog_manager.cc:1259] Loaded cluster ID: bbfbb64d808e42ef8cd83eed0dedb959
I20260926 02:56:00.729730 1488 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9: loading cluster ID for follower catalog manager: success
I20260926 02:56:00.732558 1488 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9: acquiring CA information for follower catalog manager: success
I20260926 02:56:00.734476 1488 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3b6ae4086f845dab41100f6e93c61c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:56:01.206746 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 1220
I20260926 02:56:01.208148 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:01.320633 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 1220
I20260926 02:56:01.341864 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 1294
I20260926 02:56:01.343103 1356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:01.453727 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 1294
I20260926 02:56:01.474416 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 1368
I20260926 02:56:01.475637 1430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:01.584679 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 1368
2026-09-26T02:56:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4949 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:56:01 dist-test-slave-ktng krb5kdc[1512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:56:01 dist-test-slave-ktng krb5kdc[1512](info): set up 2 sockets
Sep 26 02:56:01 dist-test-slave-ktng krb5kdc[1512](info): commencing operation
krb5kdc: starting...
W20260926 02:56:02.688645 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:56:02 dist-test-slave-ktng krb5kdc[1512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:56:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:56:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:56:02.931783 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:36239
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:36239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:56:03.265857 1525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:03.266162 1525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:03.266252 1525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:03.273484 1525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:56:03.273592 1525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:03.273638 1525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:03.273694 1525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:56:03.273737 1525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:56:03.285156 1525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:36239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:36239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:03.286463 1525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:03.287791 1525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:03.300108 1530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:03.300149 1533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:03.300191 1531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:03.301110 1525 server_base.cc:1034] running on GCE node
I20260926 02:56:03.301860 1525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:03.303432 1525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:03.304656 1525 hybrid_clock.cc:648] HybridClock initialized: now 1790391363304596 us; error 49 us; skew 500 ppm
I20260926 02:56:03.305037 1525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:03 dist-test-slave-ktng krb5kdc[1512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391363, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:03.308261 1525 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:56:03.309964 1525 webserver.cc:460] Webserver started at http://127.19.185.254:40551/ using document root <none> and password file <none>
I20260926 02:56:03.310669 1525 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:03.310824 1525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:03.311156 1525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:56:03.314203 1525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "a06a67ed9e144bddae7deab440e2ee66"
format_stamp: "Formatted at 2026-09-26 02:56:03 on dist-test-slave-ktng"
server_key: "50a9446fbd4f72b5fd9692f78222d1cc"
server_key_iv: "7b90f523fa18b313614095d36461deaa"
server_key_version: "encryptionkey@0"
I20260926 02:56:03.314967 1525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "a06a67ed9e144bddae7deab440e2ee66"
format_stamp: "Formatted at 2026-09-26 02:56:03 on dist-test-slave-ktng"
server_key: "50a9446fbd4f72b5fd9692f78222d1cc"
server_key_iv: "7b90f523fa18b313614095d36461deaa"
server_key_version: "encryptionkey@0"
I20260926 02:56:03.320432 1525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:56:03.324082 1540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:03.325522 1525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:56:03.325731 1525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "a06a67ed9e144bddae7deab440e2ee66"
format_stamp: "Formatted at 2026-09-26 02:56:03 on dist-test-slave-ktng"
server_key: "50a9446fbd4f72b5fd9692f78222d1cc"
server_key_iv: "7b90f523fa18b313614095d36461deaa"
server_key_version: "encryptionkey@0"
I20260926 02:56:03.325955 1525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:03.348353 1525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:03.352877 1525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:03.353181 1525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:03.372066 1525 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:36239
I20260926 02:56:03.372110 1591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:36239 every 8 connection(s)
I20260926 02:56:03.373435 1525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:56:03.374851 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 1525
I20260926 02:56:03.375075 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:56:03.375978 20199 external_mini_cluster.cc:1428] Setting key 7a836e459765589fd7bcb8dda808fbe6
I20260926 02:56:03.378171 1592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:56:03.389029 1592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66: Bootstrap starting.
I20260926 02:56:03.392088 1592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:03.393072 1592 log.cc:824] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:56:03 dist-test-slave-ktng krb5kdc[1512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:03.396646 1592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66: No bootstrap required, opened a new log
I20260926 02:56:03.405230 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:03.377946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46296 (local address 127.19.185.254:36239)
0926 02:56:03.378735 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:03.378747 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:03.378787 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:56:03.380586 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:03.380609 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:03.381001 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:03.381409 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:03.382908 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:03.384306 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:03.385610 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:03.386301 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:03.386520 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:03.397068 (+ 10548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:03.397081 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:03.397097 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:03.397143 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:03.400541 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:03.401533 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:03.401541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:03.401548 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:03.401728 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:03.402153 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:03.402156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:03.402158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:03.402765 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:03.402987 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:03.403559 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 02:56:03.404028 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":344,"threads_started":1}
I20260926 02:56:03.405896 1592 raft_consensus.cc:348] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06a67ed9e144bddae7deab440e2ee66" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36239 } }
I20260926 02:56:03.406347 1592 raft_consensus.cc:374] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:03.406490 1592 raft_consensus.cc:729] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06a67ed9e144bddae7deab440e2ee66, State: Initialized, Role: FOLLOWER
I20260926 02:56:03.407164 1592 consensus_queue.cc:260] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06a67ed9e144bddae7deab440e2ee66" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36239 } }
I20260926 02:56:03.407493 1592 raft_consensus.cc:388] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:56:03.407657 1592 raft_consensus.cc:482] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:56:03.407867 1592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:56:03.409677 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:56:03.410625 1592 raft_consensus.cc:504] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06a67ed9e144bddae7deab440e2ee66" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36239 } }
I20260926 02:56:03.411140 1592 leader_election.cc:302] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a06a67ed9e144bddae7deab440e2ee66; no voters:
I20260926 02:56:03.411819 1592 leader_election.cc:288] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:56:03.411983 1597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:56:03.413944 1592 sys_catalog.cc:564] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:56:03.416179 1597 raft_consensus.cc:686] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [term 1 LEADER]: Becoming Leader. State: Replica: a06a67ed9e144bddae7deab440e2ee66, State: Running, Role: LEADER
I20260926 02:56:03.416865 1597 consensus_queue.cc:237] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06a67ed9e144bddae7deab440e2ee66" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36239 } }
I20260926 02:56:03.419981 1598 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a06a67ed9e144bddae7deab440e2ee66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06a67ed9e144bddae7deab440e2ee66" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36239 } } }
I20260926 02:56:03.420071 1599 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a06a67ed9e144bddae7deab440e2ee66. Latest consensus state: current_term: 1 leader_uuid: "a06a67ed9e144bddae7deab440e2ee66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06a67ed9e144bddae7deab440e2ee66" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36239 } } }
I20260926 02:56:03.420387 1598 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [sys.catalog]: This master's current role is: LEADER
I20260926 02:56:03.420464 1599 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66 [sys.catalog]: This master's current role is: LEADER
I20260926 02:56:03.425810 1611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:56:03.430559 1611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:56:03.438810 1611 catalog_manager.cc:1347] Generated new cluster ID: 9d388752d5e3401aa75167a4c645e314
I20260926 02:56:03.438930 1611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:56:03.447712 1611 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:56:03.449020 1611 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:56:03.454725 1611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a06a67ed9e144bddae7deab440e2ee66: Generated new TSK 0
I20260926 02:56:03.455591 1611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:56:03.470114 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 1525
I20260926 02:56:03.471189 1587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:03.563459 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 1525
2026-09-26T02:56:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1974 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:56:03 dist-test-slave-ktng krb5kdc[1620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:56:03 dist-test-slave-ktng krb5kdc[1620](info): set up 2 sockets
Sep 26 02:56:03 dist-test-slave-ktng krb5kdc[1620](info): commencing operation
krb5kdc: starting...
W20260926 02:56:04.664701 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:56:04 dist-test-slave-ktng krb5kdc[1620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:56:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:56:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:56:04.901082 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:43709
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:43709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:56:05.234313 1633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:05.234616 1633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:05.234741 1633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:05.241974 1633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:56:05.242082 1633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:05.242128 1633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:05.242177 1633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:56:05.242223 1633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:56:05.251865 1633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:43709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:43709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:05.253170 1633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:05.254449 1633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:05.265982 1641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:05.266069 1638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:05.266007 1639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:05.267249 1633 server_base.cc:1034] running on GCE node
I20260926 02:56:05.268119 1633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:05.269606 1633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:05.270851 1633 hybrid_clock.cc:648] HybridClock initialized: now 1790391365270796 us; error 64 us; skew 500 ppm
I20260926 02:56:05.271238 1633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:05 dist-test-slave-ktng krb5kdc[1620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391365, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:05.274386 1633 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:56:05.275866 1633 webserver.cc:460] Webserver started at http://127.19.185.254:43559/ using document root <none> and password file <none>
I20260926 02:56:05.276568 1633 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:05.276712 1633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:05.277011 1633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:56:05.280264 1633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a"
format_stamp: "Formatted at 2026-09-26 02:56:05 on dist-test-slave-ktng"
server_key: "4bbd60f288fac3435c59598826fcb47d"
server_key_iv: "c3b7e773e4faf0e9f3af1cd5f49d3241"
server_key_version: "encryptionkey@0"
I20260926 02:56:05.281090 1633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a"
format_stamp: "Formatted at 2026-09-26 02:56:05 on dist-test-slave-ktng"
server_key: "4bbd60f288fac3435c59598826fcb47d"
server_key_iv: "c3b7e773e4faf0e9f3af1cd5f49d3241"
server_key_version: "encryptionkey@0"
I20260926 02:56:05.286242 1633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:56:05.290717 1648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:05.292098 1633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:56:05.292279 1633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a"
format_stamp: "Formatted at 2026-09-26 02:56:05 on dist-test-slave-ktng"
server_key: "4bbd60f288fac3435c59598826fcb47d"
server_key_iv: "c3b7e773e4faf0e9f3af1cd5f49d3241"
server_key_version: "encryptionkey@0"
I20260926 02:56:05.292471 1633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:05.320043 1633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:05.324397 1633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:05.324695 1633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:05.343839 1633 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:43709
I20260926 02:56:05.343888 1699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:43709 every 8 connection(s)
I20260926 02:56:05.345256 1633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:56:05.349249 1700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:56:05.353024 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 1633
I20260926 02:56:05.353250 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:56:05.353852 20199 external_mini_cluster.cc:1428] Setting key 61974ad8a2d0e969767373a20cd69e57
I20260926 02:56:05.361614 1700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Bootstrap starting.
I20260926 02:56:05.365332 1700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:05.366686 1700 log.cc:824] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:05.369987 1700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: No bootstrap required, opened a new log
Sep 26 02:56:05 dist-test-slave-ktng krb5kdc[1620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:05.379408 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:05.356037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48356 (local address 127.19.185.254:43709)
0926 02:56:05.356862 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:05.356877 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:05.356930 (+ 53us) server_negotiation.cc:407] Connection header received
0926 02:56:05.359780 (+ 2850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:05.359810 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:05.360315 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:05.360738 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:05.362098 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:05.364135 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:05.365458 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:05.366520 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:05.366723 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:05.371453 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:05.371461 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:05.371475 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:05.371511 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:05.374775 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:05.375705 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:05.375711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:05.375717 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:05.375959 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:05.376390 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:05.376395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:05.376398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:05.377049 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:05.377291 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:05.377784 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 02:56:05.378307 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":345,"threads_started":1}
I20260926 02:56:05.379715 1700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } }
I20260926 02:56:05.380122 1700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:05.380249 1700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a7c8e8a61a4e5fb8dba29d75afdd6a, State: Initialized, Role: FOLLOWER
I20260926 02:56:05.380805 1700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } }
I20260926 02:56:05.381078 1700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:56:05.381239 1700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:56:05.381422 1700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:56:05.383980 1700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } }
I20260926 02:56:05.384466 1700 leader_election.cc:302] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92a7c8e8a61a4e5fb8dba29d75afdd6a; no voters:
I20260926 02:56:05.385133 1700 leader_election.cc:288] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:56:05.385281 1705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:56:05.386404 1705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 1 LEADER]: Becoming Leader. State: Replica: 92a7c8e8a61a4e5fb8dba29d75afdd6a, State: Running, Role: LEADER
I20260926 02:56:05.386996 1705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } }
I20260926 02:56:05.387392 1700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:56:05.390151 1707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92a7c8e8a61a4e5fb8dba29d75afdd6a. Latest consensus state: current_term: 1 leader_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } } }
I20260926 02:56:05.390642 1707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: This master's current role is: LEADER
I20260926 02:56:05.390486 1706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } } }
I20260926 02:56:05.390856 1706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: This master's current role is: LEADER
I20260926 02:56:05.396178 1714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:56:05.403200 1714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:56:05.412303 1714 catalog_manager.cc:1347] Generated new cluster ID: 5f9696a1af4f4ecea524a6e5a0bc6a8c
I20260926 02:56:05.412423 1714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:56:05.436791 1714 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:56:05.438781 1714 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:56:05.451751 1714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Generated new TSK 0
I20260926 02:56:05.452597 1714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:56:05.526892 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:43709
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:56:05.861886 1726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:05.862236 1726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:05.862411 1726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:05.869674 1726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:05.869830 1726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:05.870000 1726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:56:05.879797 1726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:05.881206 1726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:05.882580 1726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:05.896764 1732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:05.896951 1734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:05.897297 1731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:05.897686 1726 server_base.cc:1034] running on GCE node
I20260926 02:56:05.898303 1726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:05.899286 1726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:05.900574 1726 hybrid_clock.cc:648] HybridClock initialized: now 1790391365900526 us; error 54 us; skew 500 ppm
I20260926 02:56:05.900928 1726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:05 dist-test-slave-ktng krb5kdc[1620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391365, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:05.904212 1726 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:56:05.905782 1726 webserver.cc:460] Webserver started at http://127.19.185.193:42981/ using document root <none> and password file <none>
I20260926 02:56:05.906487 1726 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:05.906636 1726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:05.906941 1726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:56:05.909984 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "3661d2e3efc2472292460e1da93c8d20"
format_stamp: "Formatted at 2026-09-26 02:56:05 on dist-test-slave-ktng"
server_key: "85e73736af6d66d3a4cb7c3fe04f558a"
server_key_iv: "924aac411f45b0fb428ad414088390a3"
server_key_version: "encryptionkey@0"
I20260926 02:56:05.910779 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "3661d2e3efc2472292460e1da93c8d20"
format_stamp: "Formatted at 2026-09-26 02:56:05 on dist-test-slave-ktng"
server_key: "85e73736af6d66d3a4cb7c3fe04f558a"
server_key_iv: "924aac411f45b0fb428ad414088390a3"
server_key_version: "encryptionkey@0"
I20260926 02:56:05.916119 1726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:56:05.919543 1741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:05.920836 1726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:56:05.921020 1726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "3661d2e3efc2472292460e1da93c8d20"
format_stamp: "Formatted at 2026-09-26 02:56:05 on dist-test-slave-ktng"
server_key: "85e73736af6d66d3a4cb7c3fe04f558a"
server_key_iv: "924aac411f45b0fb428ad414088390a3"
server_key_version: "encryptionkey@0"
I20260926 02:56:05.921211 1726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:05.942529 1726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:05.946864 1726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:05.947510 1726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:05.948777 1726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:05.950253 1726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:56:05.950369 1726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:05.950527 1726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:05.950600 1726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:05.983490 1726 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:40323
I20260926 02:56:05.983532 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:40323 every 8 connection(s)
I20260926 02:56:05.984761 1726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:56:05.993396 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 1726
I20260926 02:56:05.993606 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:56:05.994261 20199 external_mini_cluster.cc:1428] Setting key afcd1d1c85474cf98ee15615ca657fa0
Sep 26 02:56:05 dist-test-slave-ktng krb5kdc[1620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391365, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:06.005857 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:05.987395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:57571 (local address 127.19.185.254:43709)
0926 02:56:05.987913 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:05.987918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:05.988925 (+ 1007us) server_negotiation.cc:407] Connection header received
0926 02:56:05.990726 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:05.990732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:05.990831 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:05.990947 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:05.992499 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:05.993479 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:05.995391 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:05.995981 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:05.996211 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:06.000156 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:06.000161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:06.000165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:06.000197 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:06.002725 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:06.003532 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:06.003537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:06.003539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:06.003654 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:06.004073 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:06.004078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:06.004081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:06.004457 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:06.004642 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:06.005379 (+ 737us) server_negotiation.cc:299] Negotiation successful
0926 02:56:06.005598 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":255,"threads_started":1}
I20260926 02:56:06.007258 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:05.987958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:57571)
0926 02:56:05.988661 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:05.988698 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:05.990457 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:05.991187 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:05.991196 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:05.991756 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:56:05.992314 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:05.992345 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:05.993680 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:05.993687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:05.995210 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:05.995222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:05.996309 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:05.996318 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:05.996633 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:05.997332 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:56:05.997388 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:56:05.999920 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:56:06.002882 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:06.002892 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:06.002908 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:06.003406 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:06.003764 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:06.003770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:06.003773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:06.003956 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:06.004638 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:56:06.004645 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:56:06.004972 (+ 327us) client_negotiation.cc:769] Sending connection context
0926 02:56:06.005209 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 02:56:06.005767 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":301,"threads_started":1}
I20260926 02:56:06.009199 1854 heartbeater.cc:347] Connected to a master server at 127.19.185.254:43709
I20260926 02:56:06.009521 1854 heartbeater.cc:464] Registering TS with master...
I20260926 02:56:06.010277 1854 heartbeater.cc:511] Master 127.19.185.254:43709 requested a full tablet report, sending...
I20260926 02:56:06.012825 1665 ts_manager.cc:194] Registered new tserver with Master: 3661d2e3efc2472292460e1da93c8d20 (127.19.185.193:40323)
I20260926 02:56:06.015057 1665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:57571
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:56:06.050040 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:43709
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:56:06.376827 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:06.377099 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:06.377192 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:06.384346 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:06.384442 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:06.384567 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:56:06.394068 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:06.395351 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:06.396835 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:06.410763 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:06.410790 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:06.410852 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:06.411655 1860 server_base.cc:1034] running on GCE node
I20260926 02:56:06.412249 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:06.413185 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:06.414415 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790391366414324 us; error 82 us; skew 500 ppm
I20260926 02:56:06.414747 1860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:06 dist-test-slave-ktng krb5kdc[1620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391366, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:06.418035 1860 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:56:06.419602 1860 webserver.cc:460] Webserver started at http://127.19.185.194:41233/ using document root <none> and password file <none>
I20260926 02:56:06.420220 1860 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:06.420317 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:06.420565 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:56:06.423501 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "68cb8bdc32d24dd5a8861ba984d80a82"
format_stamp: "Formatted at 2026-09-26 02:56:06 on dist-test-slave-ktng"
server_key: "4c2df08625b0925a02694fd46a7beff1"
server_key_iv: "a638b99b7617c4e5d0377951ca017e80"
server_key_version: "encryptionkey@0"
I20260926 02:56:06.424278 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "68cb8bdc32d24dd5a8861ba984d80a82"
format_stamp: "Formatted at 2026-09-26 02:56:06 on dist-test-slave-ktng"
server_key: "4c2df08625b0925a02694fd46a7beff1"
server_key_iv: "a638b99b7617c4e5d0377951ca017e80"
server_key_version: "encryptionkey@0"
I20260926 02:56:06.429405 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 02:56:06.432943 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:06.434345 1860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:56:06.434481 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "68cb8bdc32d24dd5a8861ba984d80a82"
format_stamp: "Formatted at 2026-09-26 02:56:06 on dist-test-slave-ktng"
server_key: "4c2df08625b0925a02694fd46a7beff1"
server_key_iv: "a638b99b7617c4e5d0377951ca017e80"
server_key_version: "encryptionkey@0"
I20260926 02:56:06.434661 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:06.455895 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:06.459806 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:06.460043 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:06.460963 1860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:06.462265 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:56:06.462340 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:06.462420 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:06.462463 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:06.495863 1860 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:41747
I20260926 02:56:06.495996 1987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:41747 every 8 connection(s)
I20260926 02:56:06.497300 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:56:06.502880 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 1860
I20260926 02:56:06.503072 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:56:06.503782 20199 external_mini_cluster.cc:1428] Setting key 6607daac0f9ab870284365fe4051c5db
Sep 26 02:56:06 dist-test-slave-ktng krb5kdc[1620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391366, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:06.520169 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:06.499908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:45537 (local address 127.19.185.254:43709)
0926 02:56:06.500119 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:06.500123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:06.501251 (+ 1128us) server_negotiation.cc:407] Connection header received
0926 02:56:06.503409 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:06.503414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:06.503487 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:06.503580 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:06.505425 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:06.506322 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:06.508918 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:06.509351 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:06.509488 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:06.513679 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:06.513684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:06.513687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:06.513720 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:06.517033 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:06.517809 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:06.517818 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:06.517820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:06.517949 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:06.518377 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:06.518383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:06.518386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:06.518651 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:06.518816 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:06.519565 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 02:56:06.519806 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 02:56:06.521157 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:06.500360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:45537)
0926 02:56:06.501093 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:06.501129 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:06.503024 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:06.503869 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:06.503881 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:06.504566 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:56:06.505260 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:06.505281 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:06.506488 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:06.506494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:06.508756 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:06.508773 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:06.509587 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:06.509596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:06.509919 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:06.510683 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:56:06.510718 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:56:06.513411 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:56:06.517194 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:06.517204 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:06.517220 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:06.517661 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:06.518069 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:06.518073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:06.518077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:06.518263 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:06.518778 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:56:06.518786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:56:06.519131 (+ 345us) client_negotiation.cc:769] Sending connection context
0926 02:56:06.519475 (+ 344us) client_negotiation.cc:240] Negotiation successful
0926 02:56:06.520005 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":283,"threads_started":1}
I20260926 02:56:06.522817 1988 heartbeater.cc:347] Connected to a master server at 127.19.185.254:43709
I20260926 02:56:06.523159 1988 heartbeater.cc:464] Registering TS with master...
I20260926 02:56:06.523911 1988 heartbeater.cc:511] Master 127.19.185.254:43709 requested a full tablet report, sending...
I20260926 02:56:06.525434 1665 ts_manager.cc:194] Registered new tserver with Master: 68cb8bdc32d24dd5a8861ba984d80a82 (127.19.185.194:41747)
I20260926 02:56:06.526238 1665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:45537
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:56:06.558871 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:43709
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:56:06.887858 1993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:06.888185 1993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:06.888352 1993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:06.896010 1993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:06.896179 1993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:06.896369 1993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:56:06.906353 1993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:06.907758 1993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:06.909336 1993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:06.923542 1998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:06.923511 1999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:06.923630 2001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:06.924504 1993 server_base.cc:1034] running on GCE node
I20260926 02:56:06.925093 1993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:06.926138 1993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:06.927419 1993 hybrid_clock.cc:648] HybridClock initialized: now 1790391366927360 us; error 45 us; skew 500 ppm
I20260926 02:56:06.927774 1993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:06 dist-test-slave-ktng krb5kdc[1620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391366, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:06.931011 1993 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:56:06.932806 1993 webserver.cc:460] Webserver started at http://127.19.185.195:34853/ using document root <none> and password file <none>
I20260926 02:56:06.933526 1993 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:06.933686 1993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:06.933995 1993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:56:06.937050 1993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "832b2a46504c4d509fa1f9f0bf427de3"
format_stamp: "Formatted at 2026-09-26 02:56:06 on dist-test-slave-ktng"
server_key: "faceeda24c63262457856d6d74bd9585"
server_key_iv: "40fa638b596c4c444f8799745dac8799"
server_key_version: "encryptionkey@0"
I20260926 02:56:06.937868 1993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "832b2a46504c4d509fa1f9f0bf427de3"
format_stamp: "Formatted at 2026-09-26 02:56:06 on dist-test-slave-ktng"
server_key: "faceeda24c63262457856d6d74bd9585"
server_key_iv: "40fa638b596c4c444f8799745dac8799"
server_key_version: "encryptionkey@0"
I20260926 02:56:06.943881 1993 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:56:06.947475 2008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:06.948920 1993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 02:56:06.949103 1993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "832b2a46504c4d509fa1f9f0bf427de3"
format_stamp: "Formatted at 2026-09-26 02:56:06 on dist-test-slave-ktng"
server_key: "faceeda24c63262457856d6d74bd9585"
server_key_iv: "40fa638b596c4c444f8799745dac8799"
server_key_version: "encryptionkey@0"
I20260926 02:56:06.949313 1993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:06.979846 1993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:06.984150 1993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:06.984462 1993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:06.987077 1993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:06.988703 1993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:56:06.988804 1993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:06.988962 1993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:06.989044 1993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:07.018309 1854 heartbeater.cc:503] Master 127.19.185.254:43709 was elected leader, sending a full tablet report...
I20260926 02:56:07.023368 1993 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:39927
I20260926 02:56:07.023418 2120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:39927 every 8 connection(s)
I20260926 02:56:07.024834 1993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:56:07.031742 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 1993
I20260926 02:56:07.031956 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:56:07.032416 20199 external_mini_cluster.cc:1428] Setting key d0e4c78866490c0e7daf47475e97bfaf
Sep 26 02:56:07 dist-test-slave-ktng krb5kdc[1620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391366, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:07.045648 2123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.027366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:50431 (local address 127.19.185.254:43709)
0926 02:56:07.027835 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:07.027839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:07.028770 (+ 931us) server_negotiation.cc:407] Connection header received
0926 02:56:07.030619 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:07.030624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:07.030718 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:07.030836 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:07.032509 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.033471 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.035021 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.035882 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.036046 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.039975 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:07.039980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:07.039984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:07.040023 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:07.042531 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:07.043339 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:07.043347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:07.043355 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:07.043473 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:07.043924 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:07.043928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:07.043930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:07.044191 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:07.044354 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:07.045039 (+ 685us) server_negotiation.cc:299] Negotiation successful
0926 02:56:07.045263 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":238,"threads_started":1}
I20260926 02:56:07.046844 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.027885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:50431)
0926 02:56:07.028614 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:07.028651 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:07.030332 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:07.031084 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:07.031092 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:07.031669 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:56:07.032333 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:07.032351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.033631 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.033635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:07.034872 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:07.034884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.036134 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.036144 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:07.036416 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.037027 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:56:07.037051 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:56:07.039746 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:56:07.042701 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:07.042710 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:07.042733 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:07.043177 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:07.043591 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:07.043596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:07.043600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:07.043789 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:07.044326 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:56:07.044334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:56:07.044677 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 02:56:07.044939 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 02:56:07.045476 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":278,"threads_started":1}
I20260926 02:56:07.048472 2121 heartbeater.cc:347] Connected to a master server at 127.19.185.254:43709
I20260926 02:56:07.048768 2121 heartbeater.cc:464] Registering TS with master...
I20260926 02:56:07.049486 2121 heartbeater.cc:511] Master 127.19.185.254:43709 requested a full tablet report, sending...
I20260926 02:56:07.050892 1664 ts_manager.cc:194] Registered new tserver with Master: 832b2a46504c4d509fa1f9f0bf427de3 (127.19.185.195:39927)
I20260926 02:56:07.051882 1664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:50431
I20260926 02:56:07.060412 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:56:07.077301 2123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.064910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.19.185.254:43709)
0926 02:56:07.065179 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:07.065185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:07.065589 (+ 404us) server_negotiation.cc:407] Connection header received
0926 02:56:07.065709 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:07.065714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:07.065778 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:07.065862 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:07.067227 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.068070 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.069349 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.069784 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.069878 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.072234 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:07.072238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:07.072242 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:07.072273 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:07.074876 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:07.075540 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:07.075544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:07.075546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:07.075669 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:07.076010 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:07.076013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:07.076015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:07.076315 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:07.076505 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:07.076851 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 02:56:07.077062 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 02:56:07.085901 1664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:56:07.093120 1664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:56:07.119279 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.106020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52584 (local address 127.19.185.194:41747)
0926 02:56:07.111427 (+ 5407us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:07.111435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:07.111466 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:56:07.111564 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:07.111570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:07.111785 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:07.112689 (+ 904us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:07.112977 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.114252 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.116437 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.118081 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.118253 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.118457 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:07.118766 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 02:56:07.118944 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5224,"thread_start_us":5087,"threads_started":1}
I20260926 02:56:07.119745 2137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.106642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.19.185.193:40323)
0926 02:56:07.108251 (+ 1609us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:07.108264 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:07.110719 (+ 2455us) server_negotiation.cc:407] Connection header received
0926 02:56:07.110854 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:07.110863 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:07.111159 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:07.111401 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:07.112086 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.113672 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.116709 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.118198 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.118371 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.118527 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:07.119186 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 02:56:07.119379 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1388,"thread_start_us":1338,"threads_started":1}
I20260926 02:56:07.119967 2138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.105941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40323 (local address 127.0.0.1:33712)
0926 02:56:07.110255 (+ 4314us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:07.110283 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:07.110486 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:07.111433 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:07.111440 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:07.111469 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:07.111847 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:07.111864 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.113835 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.113840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:07.116529 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:07.116549 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.118616 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.118627 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:07.118791 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.118821 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 02:56:07.118909 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 02:56:07.119047 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4148,"mutex_wait_us":1,"thread_start_us":2517,"threads_started":1}
I20260926 02:56:07.120234 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.105816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:41747 (local address 127.0.0.1:52584)
0926 02:56:07.108872 (+ 3056us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:07.110222 (+ 1350us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:07.110473 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:07.112010 (+ 1537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:07.112016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:07.112042 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:07.112507 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:07.112515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.114390 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.114397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:07.116287 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:07.116300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.118286 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.118292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:07.118504 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.118562 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 02:56:07.118708 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 02:56:07.118910 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2753,"mutex_wait_us":25,"thread_start_us":4992,"threads_started":1}
I20260926 02:56:07.120582 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.107918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:39927 (local address 127.0.0.1:35906)
0926 02:56:07.111128 (+ 3210us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:07.111150 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:07.111251 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:07.111934 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:07.111938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:07.111968 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:07.112485 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:07.112502 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.114435 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.114441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:07.117547 (+ 3106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:07.117560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.119740 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.119749 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:07.119937 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.119974 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:56:07.120084 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 02:56:07.120216 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3048,"thread_start_us":1985,"threads_started":1}
I20260926 02:56:07.121058 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.108131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.19.185.195:39927)
0926 02:56:07.109677 (+ 1546us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:07.109689 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:07.111379 (+ 1690us) server_negotiation.cc:407] Connection header received
0926 02:56:07.111475 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:07.111482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:07.111709 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:07.111926 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:07.112689 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.114265 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.117723 (+ 3458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.119094 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.119277 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.119455 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:07.120293 (+ 838us) server_negotiation.cc:299] Negotiation successful
0926 02:56:07.120790 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1351,"thread_start_us":1202,"threads_started":1}
I20260926 02:56:07.124290 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2ae62fae2f4a3b82749c68e7cf9080 (DEFAULT_TABLE table=test-table [id=981a7a82b3bd4304a6cd64a902f3284a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:56:07.124287 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2ae62fae2f4a3b82749c68e7cf9080 (DEFAULT_TABLE table=test-table [id=981a7a82b3bd4304a6cd64a902f3284a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:56:07.124385 2056 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2ae62fae2f4a3b82749c68e7cf9080 (DEFAULT_TABLE table=test-table [id=981a7a82b3bd4304a6cd64a902f3284a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:56:07.125737 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2ae62fae2f4a3b82749c68e7cf9080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:56:07.125751 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2ae62fae2f4a3b82749c68e7cf9080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:56:07.125751 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2ae62fae2f4a3b82749c68e7cf9080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:56:07.137121 2140 tablet_bootstrap.cc:492] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3: Bootstrap starting.
I20260926 02:56:07.140223 2141 tablet_bootstrap.cc:492] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20: Bootstrap starting.
I20260926 02:56:07.141412 2140 tablet_bootstrap.cc:654] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:07.142771 2140 log.cc:824] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:07.143656 2142 tablet_bootstrap.cc:492] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82: Bootstrap starting.
I20260926 02:56:07.143855 2141 tablet_bootstrap.cc:654] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:07.145289 2141 log.cc:824] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:07.147083 2142 tablet_bootstrap.cc:654] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:07.148335 2141 tablet_bootstrap.cc:492] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20: No bootstrap required, opened a new log
I20260926 02:56:07.148313 2142 log.cc:824] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:07.148636 2141 ts_tablet_manager.cc:1397] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 02:56:07.148756 2140 tablet_bootstrap.cc:492] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3: No bootstrap required, opened a new log
I20260926 02:56:07.149080 2140 ts_tablet_manager.cc:1397] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260926 02:56:07.151243 2142 tablet_bootstrap.cc:492] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82: No bootstrap required, opened a new log
I20260926 02:56:07.151542 2142 ts_tablet_manager.cc:1397] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260926 02:56:07.156798 2141 raft_consensus.cc:348] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } } peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } }
I20260926 02:56:07.157160 2141 raft_consensus.cc:374] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:07.157282 2141 raft_consensus.cc:729] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3661d2e3efc2472292460e1da93c8d20, State: Initialized, Role: FOLLOWER
I20260926 02:56:07.157913 2141 consensus_queue.cc:260] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } } peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } }
I20260926 02:56:07.159572 2140 raft_consensus.cc:348] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } } peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } }
I20260926 02:56:07.159991 2140 raft_consensus.cc:374] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:07.160079 2140 raft_consensus.cc:729] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 832b2a46504c4d509fa1f9f0bf427de3, State: Initialized, Role: FOLLOWER
I20260926 02:56:07.160722 2141 ts_tablet_manager.cc:1428] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260926 02:56:07.160719 2140 consensus_queue.cc:260] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } } peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } }
I20260926 02:56:07.162233 2142 raft_consensus.cc:348] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } } peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } }
I20260926 02:56:07.162732 2142 raft_consensus.cc:374] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:07.162843 2142 raft_consensus.cc:729] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68cb8bdc32d24dd5a8861ba984d80a82, State: Initialized, Role: FOLLOWER
I20260926 02:56:07.163673 2142 consensus_queue.cc:260] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } } peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } }
I20260926 02:56:07.166918 1988 heartbeater.cc:503] Master 127.19.185.254:43709 was elected leader, sending a full tablet report...
I20260926 02:56:07.167657 2142 ts_tablet_manager.cc:1428] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260926 02:56:07.168455 2121 heartbeater.cc:503] Master 127.19.185.254:43709 was elected leader, sending a full tablet report...
I20260926 02:56:07.172883 2140 ts_tablet_manager.cc:1428] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3: Time spent starting tablet: real 0.024s user 0.015s sys 0.004s
I20260926 02:56:07.222739 2148 raft_consensus.cc:482] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:56:07.223089 2148 raft_consensus.cc:504] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } } peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } }
I20260926 02:56:07.224969 2148 leader_election.cc:288] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3661d2e3efc2472292460e1da93c8d20 (127.19.185.193:40323), 832b2a46504c4d509fa1f9f0bf427de3 (127.19.185.195:39927)
I20260926 02:56:07.232473 2137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.225404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:45473 (local address 127.19.185.193:40323)
0926 02:56:07.225647 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:07.225655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:07.226290 (+ 635us) server_negotiation.cc:407] Connection header received
0926 02:56:07.226537 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:07.226543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:07.226633 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:07.226738 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:07.227483 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.228359 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.230353 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.231522 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.231710 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.231791 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:07.232094 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 02:56:07.232213 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:56:07.232707 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.226035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:39927 (local address 127.19.185.194:32965)
0926 02:56:07.226775 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:07.226794 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:07.226933 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:07.227348 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:07.227352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:07.227371 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:07.227793 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:07.227808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.228903 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.228908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:07.230410 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:07.230431 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.231810 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.231816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:07.231988 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.232010 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 02:56:07.232113 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 02:56:07.232235 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":4,"thread_start_us":403,"threads_started":1}
I20260926 02:56:07.232518 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.225522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:32965 (local address 127.19.185.195:39927)
0926 02:56:07.225753 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:07.225758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:07.226889 (+ 1131us) server_negotiation.cc:407] Connection header received
0926 02:56:07.227081 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:07.227087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:07.227171 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:07.227276 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:07.227948 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.228755 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.230565 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.231585 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.231773 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.231848 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:07.232137 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 02:56:07.232257 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 02:56:07.232704 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.225330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40323 (local address 127.19.185.194:45473)
0926 02:56:07.226144 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:07.226164 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:07.226354 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:07.226777 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:07.226780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:07.226802 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:07.227225 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:07.227242 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.228508 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.228514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:07.230181 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:07.230195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.231762 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.231768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:07.231940 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.231964 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:56:07.232070 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 02:56:07.232197 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":22,"thread_start_us":342,"threads_started":1}
I20260926 02:56:07.233687 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2ae62fae2f4a3b82749c68e7cf9080" candidate_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3661d2e3efc2472292460e1da93c8d20" is_pre_election: true
I20260926 02:56:07.233701 2076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2ae62fae2f4a3b82749c68e7cf9080" candidate_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "832b2a46504c4d509fa1f9f0bf427de3" is_pre_election: true
I20260926 02:56:07.234253 2076 raft_consensus.cc:2454] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68cb8bdc32d24dd5a8861ba984d80a82 in term 0.
I20260926 02:56:07.234256 1809 raft_consensus.cc:2454] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68cb8bdc32d24dd5a8861ba984d80a82 in term 0.
I20260926 02:56:07.234885 1878 leader_election.cc:302] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68cb8bdc32d24dd5a8861ba984d80a82, 832b2a46504c4d509fa1f9f0bf427de3; no voters:
I20260926 02:56:07.235282 2148 raft_consensus.cc:2781] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:56:07.235457 2148 raft_consensus.cc:482] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:56:07.235567 2148 raft_consensus.cc:3037] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Advancing to term 1
W20260926 02:56:07.237053 1855 tablet.cc:2369] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:56:07.237869 2148 raft_consensus.cc:504] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } } peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } }
I20260926 02:56:07.238493 2148 leader_election.cc:288] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [CANDIDATE]: Term 1 election: Requested vote from peers 3661d2e3efc2472292460e1da93c8d20 (127.19.185.193:40323), 832b2a46504c4d509fa1f9f0bf427de3 (127.19.185.195:39927)
I20260926 02:56:07.238884 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2ae62fae2f4a3b82749c68e7cf9080" candidate_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3661d2e3efc2472292460e1da93c8d20"
I20260926 02:56:07.239012 2076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2ae62fae2f4a3b82749c68e7cf9080" candidate_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "832b2a46504c4d509fa1f9f0bf427de3"
I20260926 02:56:07.239077 1809 raft_consensus.cc:3037] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:56:07.239226 2076 raft_consensus.cc:3037] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:56:07.241359 1809 raft_consensus.cc:2454] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68cb8bdc32d24dd5a8861ba984d80a82 in term 1.
I20260926 02:56:07.241463 2076 raft_consensus.cc:2454] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68cb8bdc32d24dd5a8861ba984d80a82 in term 1.
I20260926 02:56:07.241793 1876 leader_election.cc:302] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3661d2e3efc2472292460e1da93c8d20, 68cb8bdc32d24dd5a8861ba984d80a82; no voters:
I20260926 02:56:07.242214 2148 raft_consensus.cc:2781] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:56:07.242650 2148 raft_consensus.cc:686] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 1 LEADER]: Becoming Leader. State: Replica: 68cb8bdc32d24dd5a8861ba984d80a82, State: Running, Role: LEADER
I20260926 02:56:07.243114 2148 consensus_queue.cc:237] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } } peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } }
I20260926 02:56:07.247790 1663 catalog_manager.cc:5495] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68cb8bdc32d24dd5a8861ba984d80a82 (127.19.185.194). New cstate: current_term: 1 leader_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } health_report { overall_health: UNKNOWN } } }
W20260926 02:56:07.249357 1989 tablet.cc:2369] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:56:07.273887 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.268512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.19.185.194:41747)
0926 02:56:07.268687 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:07.268691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:07.268710 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:56:07.268822 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:07.268827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:07.268934 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:07.269063 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:56:07.269401 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.270093 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.271593 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.272046 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.272151 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.272675 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:56:07.273285 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:56:07.273416 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:07.273531 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 02:56:07.273602 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 02:56:07.276517 2122 tablet.cc:2369] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:56:07.282594 1809 raft_consensus.cc:1261] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20 [term 1 FOLLOWER]: Refusing update from remote peer 68cb8bdc32d24dd5a8861ba984d80a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:56:07.282594 2076 raft_consensus.cc:1261] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3 [term 1 FOLLOWER]: Refusing update from remote peer 68cb8bdc32d24dd5a8861ba984d80a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:56:07.283695 2151 consensus_queue.cc:1035] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:56:07.284037 2148 consensus_queue.cc:1035] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:56:07.298439 2154 mvcc.cc:205] Tried to move back new op lower bound from 7333443040379019264 to 7333443040231391232. Current Snapshot: MvccSnapshot[applied={T|T < 7333443040379019264}]
I20260926 02:56:07.316453 1663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:56:07.316731 1663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:56:07.320721 1663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Sending DeleteTablet for 3 replicas of tablet 2d2ae62fae2f4a3b82749c68e7cf9080
I20260926 02:56:07.321806 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2ae62fae2f4a3b82749c68e7cf9080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:56:07 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:52584
I20260926 02:56:07.321943 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2ae62fae2f4a3b82749c68e7cf9080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:56:07 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:33712
I20260926 02:56:07.321892 2056 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2ae62fae2f4a3b82749c68e7cf9080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:56:07 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:35906
I20260926 02:56:07.325217 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 1726
I20260926 02:56:07.334925 2166 tablet_replica.cc:330] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20: stopping tablet replica
I20260926 02:56:07.335902 2166 raft_consensus.cc:2229] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:56:07.336467 2165 tablet_replica.cc:330] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3: stopping tablet replica
I20260926 02:56:07.336683 2164 tablet_replica.cc:330] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82: stopping tablet replica
I20260926 02:56:07.336669 2166 raft_consensus.cc:2258] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:56:07.336943 2165 raft_consensus.cc:2229] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:56:07.337081 2164 raft_consensus.cc:2229] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:56:07.337461 2165 raft_consensus.cc:2258] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:56:07.337817 2164 raft_consensus.cc:2258] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:56:07.339643 2166 ts_tablet_manager.cc:1905] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:56:07.340029 2164 ts_tablet_manager.cc:1905] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:56:07.341017 2165 ts_tablet_manager.cc:1905] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:56:07.347744 2164 ts_tablet_manager.cc:1918] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:56:07.347956 2164 log.cc:1196] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/2d2ae62fae2f4a3b82749c68e7cf9080
I20260926 02:56:07.348520 2164 ts_tablet_manager.cc:1939] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 68cb8bdc32d24dd5a8861ba984d80a82: Deleting consensus metadata
I20260926 02:56:07.350442 1652 catalog_manager.cc:4841] TS 68cb8bdc32d24dd5a8861ba984d80a82 (127.19.185.194:41747): tablet 2d2ae62fae2f4a3b82749c68e7cf9080 (table test-table [id=981a7a82b3bd4304a6cd64a902f3284a]) successfully deleted
I20260926 02:56:07.351984 2165 ts_tablet_manager.cc:1918] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:56:07.352164 2165 log.cc:1196] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/2d2ae62fae2f4a3b82749c68e7cf9080
I20260926 02:56:07.352727 2165 ts_tablet_manager.cc:1939] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 832b2a46504c4d509fa1f9f0bf427de3: Deleting consensus metadata
Sep 26 02:56:07 dist-test-slave-ktng krb5kdc[1620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:56:07.354674 1651 catalog_manager.cc:4841] TS 832b2a46504c4d509fa1f9f0bf427de3 (127.19.185.195:39927): tablet 2d2ae62fae2f4a3b82749c68e7cf9080 (table test-table [id=981a7a82b3bd4304a6cd64a902f3284a]) successfully deleted
I20260926 02:56:07.358309 2166 ts_tablet_manager.cc:1918] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:56:07.358568 2166 log.cc:1196] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/2d2ae62fae2f4a3b82749c68e7cf9080
I20260926 02:56:07.359225 2166 ts_tablet_manager.cc:1939] T 2d2ae62fae2f4a3b82749c68e7cf9080 P 3661d2e3efc2472292460e1da93c8d20: Deleting consensus metadata
I20260926 02:56:07.361202 1649 catalog_manager.cc:4841] TS 3661d2e3efc2472292460e1da93c8d20 (127.19.185.193:40323): tablet 2d2ae62fae2f4a3b82749c68e7cf9080 (table test-table [id=981a7a82b3bd4304a6cd64a902f3284a]) successfully deleted
I20260926 02:56:07.363554 2137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.326145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.19.185.193:40323)
0926 02:56:07.326324 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:07.326330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:07.331794 (+ 5464us) server_negotiation.cc:407] Connection header received
0926 02:56:07.332938 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:07.332947 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:07.333057 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:07.333161 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:07.338010 (+ 4849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.339174 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.345602 (+ 6428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.346255 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.346358 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.355715 (+ 9357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:07.355730 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:07.355752 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:07.355805 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:07.359865 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:07.360771 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:07.360780 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:07.360783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:07.360954 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:07.361397 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:07.361402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:07.361405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:07.361844 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:07.362141 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:07.362749 (+ 608us) server_negotiation.cc:299] Negotiation successful
0926 02:56:07.363075 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:56:07.364447 1849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:07.456300 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 1726
I20260926 02:56:07.474066 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 1860
Sep 26 02:56:07 dist-test-slave-ktng krb5kdc[1620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:56:07.493817 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.474879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.19.185.194:41747)
0926 02:56:07.475412 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:07.475416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:07.475432 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:56:07.475485 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:07.475489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:07.475562 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:07.475641 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:07.477343 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.478224 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.480209 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.480655 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.480758 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.486153 (+ 5395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:07.486176 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:07.486193 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:07.486243 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:07.490077 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:07.491272 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:07.491280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:07.491282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:07.491523 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:07.492009 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:07.492013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:07.492015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:07.492390 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:07.492659 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:07.493204 (+ 545us) server_negotiation.cc:299] Negotiation successful
0926 02:56:07.493449 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355}
I20260926 02:56:07.494658 1983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:07.616832 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 1860
I20260926 02:56:07.635360 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 1993
Sep 26 02:56:07 dist-test-slave-ktng krb5kdc[1620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:56:07.650573 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:07.636147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.19.185.195:39927)
0926 02:56:07.636322 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:07.636329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:07.636349 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:56:07.636464 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:07.636468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:07.636545 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:07.636656 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:07.637961 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.638715 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.640088 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:07.640496 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:07.640635 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:07.644950 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:07.644964 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:07.644979 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:07.645012 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:07.647844 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:07.648533 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:07.648541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:07.648543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:07.648673 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:07.649053 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:07.649062 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:07.649064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:07.649399 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:07.649720 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:07.649998 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 02:56:07.650223 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:56:07.651222 2116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:07.741478 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 1993
I20260926 02:56:07.758029 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 1633
I20260926 02:56:07.759423 1695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:07.873360 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 1633
I20260926 02:56:07.894887 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:43709
--webserver_interface=127.19.185.254
--webserver_port=43559
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:43709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:56:08.229187 2172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:08.229483 2172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:08.229655 2172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:08.236775 2172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:56:08.236878 2172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:08.236999 2172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:08.237105 2172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:56:08.237185 2172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:56:08.246923 2172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:43709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:43709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=43559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:08.248234 2172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:08.249557 2172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:08.273397 2181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:08.273469 2178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:08.273468 2179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:08.274233 2172 server_base.cc:1034] running on GCE node
I20260926 02:56:08.275146 2172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:08.276926 2172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:08.278183 2172 hybrid_clock.cc:648] HybridClock initialized: now 1790391368278116 us; error 53 us; skew 500 ppm
I20260926 02:56:08.278618 2172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:08 dist-test-slave-ktng krb5kdc[1620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391368, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:08.282269 2172 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:56:08.283994 2172 webserver.cc:460] Webserver started at http://127.19.185.254:43559/ using document root <none> and password file <none>
I20260926 02:56:08.284780 2172 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:08.284945 2172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:08.291005 2172 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 02:56:08.294040 2188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:08.295550 2172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:56:08.295746 2172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a"
format_stamp: "Formatted at 2026-09-26 02:56:05 on dist-test-slave-ktng"
server_key: "4bbd60f288fac3435c59598826fcb47d"
server_key_iv: "c3b7e773e4faf0e9f3af1cd5f49d3241"
server_key_version: "encryptionkey@0"
I20260926 02:56:08.296923 2172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:08.328510 2172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:08.333253 2172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:08.333534 2172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:08.355971 2172 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:43709
I20260926 02:56:08.355994 2239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:43709 every 8 connection(s)
I20260926 02:56:08.357741 2172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:56:08.358448 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 2172
I20260926 02:56:08.359629 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:40323
--local_ip_for_outbound_sockets=127.19.185.193
--tserver_master_addrs=127.19.185.254:43709
--webserver_port=42981
--webserver_interface=127.19.185.193
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:56:08.371933 2240 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:56:08.377405 2240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Bootstrap starting.
I20260926 02:56:08.405719 2240 log.cc:824] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:08.417694 2240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:56:08.418256 2240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Bootstrap complete.
I20260926 02:56:08.428290 2240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } }
I20260926 02:56:08.429164 2240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a7c8e8a61a4e5fb8dba29d75afdd6a, State: Initialized, Role: FOLLOWER
I20260926 02:56:08.429814 2240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } }
I20260926 02:56:08.430081 2240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:56:08.430225 2240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:56:08.430406 2240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:56:08.434434 2240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } }
I20260926 02:56:08.434962 2240 leader_election.cc:302] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92a7c8e8a61a4e5fb8dba29d75afdd6a; no voters:
I20260926 02:56:08.435660 2240 leader_election.cc:288] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:56:08.435803 2244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:56:08.437203 2244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 2 LEADER]: Becoming Leader. State: Replica: 92a7c8e8a61a4e5fb8dba29d75afdd6a, State: Running, Role: LEADER
I20260926 02:56:08.437783 2244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } }
I20260926 02:56:08.438158 2240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:56:08.440742 2246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92a7c8e8a61a4e5fb8dba29d75afdd6a. Latest consensus state: current_term: 2 leader_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } } }
I20260926 02:56:08.441202 2246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: This master's current role is: LEADER
I20260926 02:56:08.441764 2245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } } }
I20260926 02:56:08.442090 2245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: This master's current role is: LEADER
I20260926 02:56:08.451972 2251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:56:08.461436 2251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:56:08.462994 2251 catalog_manager.cc:1259] Loaded cluster ID: 5f9696a1af4f4ecea524a6e5a0bc6a8c
I20260926 02:56:08.463208 2251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:56:08.466276 2251 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 02:56:08.484025 2251 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 02:56:08.493687 2251 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:56:08.495536 2251 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Loaded TSK: 0
I20260926 02:56:08.497093 2251 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:56:08.728245 2242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:08.728528 2242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:08.728680 2242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:08.736109 2242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:08.736248 2242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:08.736430 2242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:56:08.746788 2242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:40323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=42981
--tserver_master_addrs=127.19.185.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:08.748194 2242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:08.749537 2242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:08.763757 2267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:08.763916 2270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:08.763972 2268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:08.765798 2242 server_base.cc:1034] running on GCE node
I20260926 02:56:08.766479 2242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:08.767570 2242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:08.768810 2242 hybrid_clock.cc:648] HybridClock initialized: now 1790391368768747 us; error 54 us; skew 500 ppm
I20260926 02:56:08.769186 2242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:08 dist-test-slave-ktng krb5kdc[1620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391368, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:08.777452 2242 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:56:08.779413 2242 webserver.cc:460] Webserver started at http://127.19.185.193:42981/ using document root <none> and password file <none>
I20260926 02:56:08.780368 2242 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:08.780524 2242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:08.787663 2242 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 02:56:08.791047 2277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:08.792968 2242 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:56:08.793146 2242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "3661d2e3efc2472292460e1da93c8d20"
format_stamp: "Formatted at 2026-09-26 02:56:05 on dist-test-slave-ktng"
server_key: "85e73736af6d66d3a4cb7c3fe04f558a"
server_key_iv: "924aac411f45b0fb428ad414088390a3"
server_key_version: "encryptionkey@0"
I20260926 02:56:08.794493 2242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:08.831396 2242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:08.836921 2242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:08.837211 2242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:08.838500 2242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:08.840435 2242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:56:08.840543 2242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:08.840649 2242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:08.840723 2242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:08.874641 2242 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:40323
I20260926 02:56:08.874714 2389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:40323 every 8 connection(s)
I20260926 02:56:08.876178 2242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:56:08.879462 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 2242
I20260926 02:56:08.880328 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:41747
--local_ip_for_outbound_sockets=127.19.185.194
--tserver_master_addrs=127.19.185.254:43709
--webserver_port=41233
--webserver_interface=127.19.185.194
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:56:08 dist-test-slave-ktng krb5kdc[1620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391368, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:08.905097 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:08.880118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:43387 (local address 127.19.185.254:43709)
0926 02:56:08.880704 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:08.880719 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:08.880858 (+ 139us) server_negotiation.cc:407] Connection header received
0926 02:56:08.884086 (+ 3228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:08.884100 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:08.884452 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:08.884802 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:08.886879 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:08.888308 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:08.889942 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:08.891076 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:08.891271 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:08.895419 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:08.895434 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:08.895457 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:08.895518 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:08.899736 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:08.900636 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:08.900647 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:08.900659 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:08.900888 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:08.901396 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:08.901401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:08.901405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:08.902250 (+ 845us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:08.902496 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:08.903330 (+ 834us) server_negotiation.cc:299] Negotiation successful
0926 02:56:08.903724 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":939,"threads_started":1}
I20260926 02:56:08.905097 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:08.879549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:43387)
0926 02:56:08.880663 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:08.880715 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:08.883487 (+ 2772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:08.885010 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:08.885023 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:08.885832 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:56:08.886627 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:08.886658 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:08.888502 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:08.888506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:08.889785 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:08.889798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:08.891572 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:08.891580 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:08.891782 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:08.892563 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:56:08.892586 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:56:08.895070 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:56:08.899931 (+ 4861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:08.899938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:08.899950 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:08.900440 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:08.901028 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:08.901031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:08.901034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:08.901217 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:08.902420 (+ 1203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:56:08.902428 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:56:08.902777 (+ 349us) client_negotiation.cc:769] Sending connection context
0926 02:56:08.903096 (+ 319us) client_negotiation.cc:240] Negotiation successful
0926 02:56:08.903689 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":636,"threads_started":1}
I20260926 02:56:08.908093 2390 heartbeater.cc:347] Connected to a master server at 127.19.185.254:43709
I20260926 02:56:08.908427 2390 heartbeater.cc:464] Registering TS with master...
I20260926 02:56:08.909148 2390 heartbeater.cc:511] Master 127.19.185.254:43709 requested a full tablet report, sending...
I20260926 02:56:08.911368 2205 ts_manager.cc:194] Registered new tserver with Master: 3661d2e3efc2472292460e1da93c8d20 (127.19.185.193:40323)
I20260926 02:56:08.912839 2205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:43387
W20260926 02:56:09.233559 2394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:09.234094 2394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:09.234349 2394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:09.246642 2394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:09.246824 2394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:09.247036 2394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:56:09.266844 2394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:41747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=41233
--tserver_master_addrs=127.19.185.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:09.268319 2394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:09.269717 2394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:09.282994 2399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:09.282949 2402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:09.283078 2400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:09.284193 2394 server_base.cc:1034] running on GCE node
I20260926 02:56:09.284879 2394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:09.285923 2394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:09.287153 2394 hybrid_clock.cc:648] HybridClock initialized: now 1790391369287097 us; error 46 us; skew 500 ppm
I20260926 02:56:09.287575 2394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:09 dist-test-slave-ktng krb5kdc[1620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391369, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:09.290882 2394 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:56:09.292480 2394 webserver.cc:460] Webserver started at http://127.19.185.194:41233/ using document root <none> and password file <none>
I20260926 02:56:09.293253 2394 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:09.293412 2394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:09.298985 2394 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 02:56:09.302270 2409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:09.304180 2394 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 02:56:09.304356 2394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "68cb8bdc32d24dd5a8861ba984d80a82"
format_stamp: "Formatted at 2026-09-26 02:56:06 on dist-test-slave-ktng"
server_key: "4c2df08625b0925a02694fd46a7beff1"
server_key_iv: "a638b99b7617c4e5d0377951ca017e80"
server_key_version: "encryptionkey@0"
I20260926 02:56:09.305438 2394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:09.324991 2394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:09.328899 2394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:09.329176 2394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:09.330147 2394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:09.331545 2394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:56:09.331664 2394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:09.331802 2394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:09.331895 2394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:09.367219 2394 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:41747
I20260926 02:56:09.367455 2521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:41747 every 8 connection(s)
I20260926 02:56:09.368949 2394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:56:09.369547 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 2394
I20260926 02:56:09.370311 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:39927
--local_ip_for_outbound_sockets=127.19.185.195
--tserver_master_addrs=127.19.185.254:43709
--webserver_port=34853
--webserver_interface=127.19.185.195
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:56:09 dist-test-slave-ktng krb5kdc[1620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391369, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:09.396667 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:09.373864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:54431 (local address 127.19.185.254:43709)
0926 02:56:09.374103 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:09.374112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:09.380003 (+ 5891us) server_negotiation.cc:407] Connection header received
0926 02:56:09.381849 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:09.381855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:09.381960 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:09.382078 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:09.383680 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.384374 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.385963 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.386378 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.386526 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:09.390598 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:09.390602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:09.390606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:09.390641 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:09.393256 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:09.394250 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:09.394254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:09.394256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:09.394376 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:09.394908 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:09.394911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:09.394913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:09.395175 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:09.395283 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:09.396252 (+ 969us) server_negotiation.cc:299] Negotiation successful
0926 02:56:09.396420 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:56:09.398273 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:09.373171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:54431)
0926 02:56:09.379826 (+ 6655us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:09.379865 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:09.381578 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:09.382326 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:09.382335 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:09.382851 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:56:09.383508 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:09.383528 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.384540 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.384551 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:09.385802 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:09.385814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.386630 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.386637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:09.386866 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:09.387606 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:56:09.387633 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:56:09.390341 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:56:09.393405 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:09.393415 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:09.393435 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:09.394124 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:09.394490 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:09.394498 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:09.394504 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:09.394775 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:09.395325 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:56:09.395334 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:56:09.395842 (+ 508us) client_negotiation.cc:769] Sending connection context
0926 02:56:09.396163 (+ 321us) client_negotiation.cc:240] Negotiation successful
0926 02:56:09.396706 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6382,"thread_start_us":6182,"threads_started":1}
I20260926 02:56:09.399785 2522 heartbeater.cc:347] Connected to a master server at 127.19.185.254:43709
I20260926 02:56:09.400243 2522 heartbeater.cc:464] Registering TS with master...
I20260926 02:56:09.401010 2522 heartbeater.cc:511] Master 127.19.185.254:43709 requested a full tablet report, sending...
I20260926 02:56:09.402714 2205 ts_manager.cc:194] Registered new tserver with Master: 68cb8bdc32d24dd5a8861ba984d80a82 (127.19.185.194:41747)
I20260926 02:56:09.403908 2205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:54431
W20260926 02:56:09.706139 2525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:09.706419 2525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:09.706512 2525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:09.713743 2525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:09.713855 2525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:09.713964 2525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:56:09.723814 2525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:39927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=34853
--tserver_master_addrs=127.19.185.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:09.725270 2525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:09.726712 2525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:09.741119 2531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:09.741257 2530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:09.741269 2533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:09.741837 2525 server_base.cc:1034] running on GCE node
I20260926 02:56:09.742465 2525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:09.743497 2525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:09.744755 2525 hybrid_clock.cc:648] HybridClock initialized: now 1790391369744710 us; error 52 us; skew 500 ppm
I20260926 02:56:09.745113 2525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:09 dist-test-slave-ktng krb5kdc[1620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391369, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:09.748376 2525 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:56:09.749758 2525 webserver.cc:460] Webserver started at http://127.19.185.195:34853/ using document root <none> and password file <none>
I20260926 02:56:09.750485 2525 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:09.750654 2525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:09.756713 2525 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 02:56:09.759945 2540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:09.761420 2525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:56:09.761591 2525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "832b2a46504c4d509fa1f9f0bf427de3"
format_stamp: "Formatted at 2026-09-26 02:56:06 on dist-test-slave-ktng"
server_key: "faceeda24c63262457856d6d74bd9585"
server_key_iv: "40fa638b596c4c444f8799745dac8799"
server_key_version: "encryptionkey@0"
I20260926 02:56:09.762730 2525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:09.774938 2525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:09.779279 2525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:09.779619 2525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:09.780673 2525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:09.782003 2525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:56:09.782090 2525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:09.782236 2525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:09.782308 2525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:09.815887 2525 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:39927
I20260926 02:56:09.816001 2652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:39927 every 8 connection(s)
I20260926 02:56:09.817507 2525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:56:09.826944 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 2525
Sep 26 02:56:09 dist-test-slave-ktng krb5kdc[1620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391369, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:09.840780 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:09.820457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:36805 (local address 127.19.185.254:43709)
0926 02:56:09.820686 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:09.820691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:09.822038 (+ 1347us) server_negotiation.cc:407] Connection header received
0926 02:56:09.824025 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:09.824031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:09.824125 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:09.824237 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:09.825779 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.826524 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.828680 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.829397 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.829536 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:09.833822 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:09.833923 (+ 101us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:09.833929 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:09.833971 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:09.837008 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:09.838111 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:09.838118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:09.838121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:09.838293 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:09.838774 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:09.838781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:09.838786 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:09.839202 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:09.839381 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:09.840357 (+ 976us) server_negotiation.cc:299] Negotiation successful
0926 02:56:09.840536 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 02:56:09.842223 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:09.820975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:36805)
0926 02:56:09.821828 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:09.821881 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:09.823733 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:09.824494 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:09.824504 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:09.825034 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:56:09.825607 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:09.825624 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.826691 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.826701 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:09.828485 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:09.828509 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.829627 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.829635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:09.829864 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:09.830584 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:56:09.830606 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:56:09.833393 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:56:09.837170 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:09.837182 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:09.837206 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:09.837974 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:09.838407 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:09.838411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:09.838413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:09.838630 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:09.839383 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:56:09.839396 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:56:09.839896 (+ 500us) client_negotiation.cc:769] Sending connection context
0926 02:56:09.840246 (+ 350us) client_negotiation.cc:240] Negotiation successful
0926 02:56:09.840850 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":289,"threads_started":1}
I20260926 02:56:09.843883 2653 heartbeater.cc:347] Connected to a master server at 127.19.185.254:43709
I20260926 02:56:09.844280 2653 heartbeater.cc:464] Registering TS with master...
I20260926 02:56:09.844385 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:09.828016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.19.185.254:43709)
0926 02:56:09.828837 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:09.828842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:09.828879 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:56:09.828922 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:09.828926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:09.829019 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:09.829126 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:09.830926 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.831874 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.833777 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.834563 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.834698 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:09.837958 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:09.837964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:09.837968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:09.838001 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:09.841522 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:09.842199 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:09.842206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:09.842212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:09.842354 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:09.842786 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:09.842791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:09.842796 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:09.843173 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:09.843462 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:09.843770 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 02:56:09.843982 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":579,"threads_started":1}
I20260926 02:56:09.845079 2653 heartbeater.cc:511] Master 127.19.185.254:43709 requested a full tablet report, sending...
I20260926 02:56:09.846635 2204 ts_manager.cc:194] Registered new tserver with Master: 832b2a46504c4d509fa1f9f0bf427de3 (127.19.185.195:39927)
I20260926 02:56:09.847617 2204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:36805
I20260926 02:56:09.859501 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:56:09.866550 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:09.860477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.19.185.254:43709)
0926 02:56:09.860716 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:09.860725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:09.861150 (+ 425us) server_negotiation.cc:407] Connection header received
0926 02:56:09.861320 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:09.861324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:09.861466 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:09.861594 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:56:09.862062 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.862797 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.864295 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.864742 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.864840 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:09.865395 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:56:09.865930 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:56:09.866039 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:09.866136 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 02:56:09.866202 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:56:09.867511 2204 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:56:09.874846 2204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:56:09.911926 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:09.887432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.19.185.193:40323)
0926 02:56:09.902866 (+ 15434us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:09.902876 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:09.902908 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:56:09.903017 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:09.903025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:09.903428 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:09.903645 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:09.904286 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.905667 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.908941 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.910225 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.910447 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:09.910629 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:09.911384 (+ 755us) server_negotiation.cc:299] Negotiation successful
0926 02:56:09.911555 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15265,"thread_start_us":15129,"threads_started":1}
I20260926 02:56:09.911927 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:09.888109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.19.185.195:39927)
0926 02:56:09.901895 (+ 13786us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:09.901915 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:09.902469 (+ 554us) server_negotiation.cc:407] Connection header received
0926 02:56:09.902621 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:09.902634 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:09.902996 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:09.903230 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:09.904263 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.906093 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.908907 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.910291 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.910479 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:09.910627 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:09.911408 (+ 781us) server_negotiation.cc:299] Negotiation successful
0926 02:56:09.911569 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13107,"thread_start_us":14000,"threads_started":1}
I20260926 02:56:09.911959 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:09.887195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40323 (local address 127.0.0.1:33730)
0926 02:56:09.896855 (+ 9660us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:09.896930 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:09.897154 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:09.903722 (+ 6568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:09.903727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:09.903750 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:09.904124 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:09.904138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.905826 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.905835 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:09.908774 (+ 2939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:09.908783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.910856 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.910866 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:09.911081 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:09.911150 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 02:56:09.911299 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 02:56:09.911520 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9315,"mutex_wait_us":220,"thread_start_us":15619,"threads_started":1}
I20260926 02:56:09.912232 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:09.890287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.19.185.194:41747)
0926 02:56:09.897863 (+ 7576us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:09.897876 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:09.903638 (+ 5762us) server_negotiation.cc:407] Connection header received
0926 02:56:09.903882 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:09.903892 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:09.904145 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:09.904323 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:09.905213 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.906674 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.908714 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.910301 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.910459 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:09.910627 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:09.911746 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0926 02:56:09.911898 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7305,"thread_start_us":7456,"threads_started":1}
I20260926 02:56:09.912734 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:09.889530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:41747 (local address 127.0.0.1:52618)
0926 02:56:09.903491 (+ 13961us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:09.903509 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:09.903644 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:09.904590 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:09.904595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:09.904619 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:09.905018 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:09.905030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.906821 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.906828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:09.908566 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:09.908578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.911080 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.911088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:09.911321 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:09.911356 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:56:09.911441 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 02:56:09.911568 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13819,"mutex_wait_us":51,"thread_start_us":6795,"threads_started":1}
I20260926 02:56:09.913185 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:09.887244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:39927 (local address 127.0.0.1:35922)
0926 02:56:09.902102 (+ 14858us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:09.902128 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:09.902296 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:09.903409 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:09.903417 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:09.903453 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:09.903986 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:09.904005 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.906267 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.906271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:09.908740 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:09.908761 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:09.910892 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:09.910903 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:09.911160 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:09.911196 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:56:09.911274 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 02:56:09.912784 (+ 1510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14670,"mutex_wait_us":151,"spinlock_wait_cycles":3968,"thread_start_us":15948,"threads_started":1}
I20260926 02:56:09.915796 2390 heartbeater.cc:503] Master 127.19.185.254:43709 was elected leader, sending a full tablet report...
I20260926 02:56:09.915777 2325 tablet_service.cc:1467] Processing CreateTablet for tablet 33b7c39ddc2a4ddd98829ee32a1f0a25 (DEFAULT_TABLE table=test-table [id=8164b82db80a46eb95ccf11796fdca67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:56:09.917402 2325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b7c39ddc2a4ddd98829ee32a1f0a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:56:09.916682 2457 tablet_service.cc:1467] Processing CreateTablet for tablet 33b7c39ddc2a4ddd98829ee32a1f0a25 (DEFAULT_TABLE table=test-table [id=8164b82db80a46eb95ccf11796fdca67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:56:09.918439 2457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b7c39ddc2a4ddd98829ee32a1f0a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:56:09.919039 2588 tablet_service.cc:1467] Processing CreateTablet for tablet 33b7c39ddc2a4ddd98829ee32a1f0a25 (DEFAULT_TABLE table=test-table [id=8164b82db80a46eb95ccf11796fdca67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:56:09.920781 2588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b7c39ddc2a4ddd98829ee32a1f0a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:56:09.931656 2669 tablet_bootstrap.cc:492] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82: Bootstrap starting.
I20260926 02:56:09.935912 2669 tablet_bootstrap.cc:654] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:09.937152 2669 log.cc:824] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:09.937984 2671 tablet_bootstrap.cc:492] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3: Bootstrap starting.
I20260926 02:56:09.939133 2670 tablet_bootstrap.cc:492] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: Bootstrap starting.
I20260926 02:56:09.942766 2670 tablet_bootstrap.cc:654] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:09.943256 2671 tablet_bootstrap.cc:654] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:09.944089 2670 log.cc:824] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:09.945283 2671 log.cc:824] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:09.946756 2669 tablet_bootstrap.cc:492] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82: No bootstrap required, opened a new log
I20260926 02:56:09.947098 2669 ts_tablet_manager.cc:1397] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260926 02:56:09.948544 2671 tablet_bootstrap.cc:492] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3: No bootstrap required, opened a new log
I20260926 02:56:09.948720 2670 tablet_bootstrap.cc:492] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: No bootstrap required, opened a new log
I20260926 02:56:09.948841 2671 ts_tablet_manager.cc:1397] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 02:56:09.948982 2670 ts_tablet_manager.cc:1397] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 02:56:09.955647 2669 raft_consensus.cc:348] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } } peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } }
I20260926 02:56:09.956086 2669 raft_consensus.cc:374] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:09.956195 2669 raft_consensus.cc:729] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68cb8bdc32d24dd5a8861ba984d80a82, State: Initialized, Role: FOLLOWER
I20260926 02:56:09.956961 2669 consensus_queue.cc:260] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } } peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } }
I20260926 02:56:09.959613 2522 heartbeater.cc:503] Master 127.19.185.254:43709 was elected leader, sending a full tablet report...
I20260926 02:56:09.959899 2671 raft_consensus.cc:348] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } } peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } }
I20260926 02:56:09.959904 2670 raft_consensus.cc:348] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } } peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } }
I20260926 02:56:09.960338 2670 raft_consensus.cc:374] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:09.960338 2671 raft_consensus.cc:374] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:09.960433 2670 raft_consensus.cc:729] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3661d2e3efc2472292460e1da93c8d20, State: Initialized, Role: FOLLOWER
I20260926 02:56:09.960445 2671 raft_consensus.cc:729] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 832b2a46504c4d509fa1f9f0bf427de3, State: Initialized, Role: FOLLOWER
I20260926 02:56:09.960698 2669 ts_tablet_manager.cc:1428] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 02:56:09.961180 2671 consensus_queue.cc:260] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } } peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } }
I20260926 02:56:09.961179 2670 consensus_queue.cc:260] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } } peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } }
I20260926 02:56:09.963061 2653 heartbeater.cc:503] Master 127.19.185.254:43709 was elected leader, sending a full tablet report...
I20260926 02:56:09.963249 2670 ts_tablet_manager.cc:1428] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 02:56:09.963553 2671 ts_tablet_manager.cc:1428] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260926 02:56:10.069267 2654 tablet.cc:2369] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:56:10.084501 2675 raft_consensus.cc:482] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:56:10.084864 2675 raft_consensus.cc:504] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } } peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } }
I20260926 02:56:10.086913 2675 leader_election.cc:288] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3661d2e3efc2472292460e1da93c8d20 (127.19.185.193:40323), 832b2a46504c4d509fa1f9f0bf427de3 (127.19.185.195:39927)
I20260926 02:56:10.096565 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.087592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:57045 (local address 127.19.185.195:39927)
0926 02:56:10.087794 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:10.087801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:10.088627 (+ 826us) server_negotiation.cc:407] Connection header received
0926 02:56:10.088811 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:10.088815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:10.088886 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:10.088992 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:10.089858 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.091020 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.094059 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.095331 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.095487 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:10.095562 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:10.096176 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 02:56:10.096308 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:56:10.096568 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.087405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:58021 (local address 127.19.185.193:40323)
0926 02:56:10.087615 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:10.087623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:10.088600 (+ 977us) server_negotiation.cc:407] Connection header received
0926 02:56:10.088785 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:10.088790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:10.088963 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:10.089058 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:10.089807 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.091020 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.094059 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.095317 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.095487 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:10.095562 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:10.096176 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 02:56:10.096308 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:56:10.096974 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.087451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:39927 (local address 127.19.185.194:57045)
0926 02:56:10.088520 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:10.088536 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:10.088685 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:10.089056 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:10.089060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:10.089088 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:10.089693 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:10.089705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.091155 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.091160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:10.093883 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:10.093899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.095623 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.095629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:10.095890 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:10.095933 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 02:56:10.096011 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 02:56:10.096113 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":230,"spinlock_wait_cycles":36096,"thread_start_us":573,"threads_started":1}
I20260926 02:56:10.096992 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.087187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40323 (local address 127.19.185.194:58021)
0926 02:56:10.088460 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:10.088481 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:10.088642 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:10.089072 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:10.089075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:10.089109 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:10.089621 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:10.089640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.091157 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.091161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:10.093883 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:10.093899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.095611 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.095618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:10.095825 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:10.095861 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:56:10.095973 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 02:56:10.096113 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"mutex_wait_us":138,"spinlock_wait_cycles":3968,"thread_start_us":634,"threads_started":1}
I20260926 02:56:10.098249 2345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b7c39ddc2a4ddd98829ee32a1f0a25" candidate_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3661d2e3efc2472292460e1da93c8d20" is_pre_election: true
I20260926 02:56:10.098238 2608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b7c39ddc2a4ddd98829ee32a1f0a25" candidate_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "832b2a46504c4d509fa1f9f0bf427de3" is_pre_election: true
I20260926 02:56:10.098851 2608 raft_consensus.cc:2454] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68cb8bdc32d24dd5a8861ba984d80a82 in term 0.
I20260926 02:56:10.098851 2345 raft_consensus.cc:2454] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68cb8bdc32d24dd5a8861ba984d80a82 in term 0.
I20260926 02:56:10.099666 2410 leader_election.cc:302] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3661d2e3efc2472292460e1da93c8d20, 68cb8bdc32d24dd5a8861ba984d80a82; no voters:
I20260926 02:56:10.100108 2675 raft_consensus.cc:2781] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:56:10.100284 2675 raft_consensus.cc:482] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:56:10.100430 2675 raft_consensus.cc:3037] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:56:10.102933 2675 raft_consensus.cc:504] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } } peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } }
I20260926 02:56:10.103688 2675 leader_election.cc:288] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [CANDIDATE]: Term 1 election: Requested vote from peers 3661d2e3efc2472292460e1da93c8d20 (127.19.185.193:40323), 832b2a46504c4d509fa1f9f0bf427de3 (127.19.185.195:39927)
I20260926 02:56:10.104177 2345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b7c39ddc2a4ddd98829ee32a1f0a25" candidate_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3661d2e3efc2472292460e1da93c8d20"
I20260926 02:56:10.104413 2345 raft_consensus.cc:3037] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:56:10.104434 2608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b7c39ddc2a4ddd98829ee32a1f0a25" candidate_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "832b2a46504c4d509fa1f9f0bf427de3"
I20260926 02:56:10.104688 2608 raft_consensus.cc:3037] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:56:10.106657 2345 raft_consensus.cc:2454] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68cb8bdc32d24dd5a8861ba984d80a82 in term 1.
I20260926 02:56:10.106943 2608 raft_consensus.cc:2454] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68cb8bdc32d24dd5a8861ba984d80a82 in term 1.
I20260926 02:56:10.107122 2410 leader_election.cc:302] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3661d2e3efc2472292460e1da93c8d20, 68cb8bdc32d24dd5a8861ba984d80a82; no voters:
I20260926 02:56:10.107450 2675 raft_consensus.cc:2781] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:56:10.107925 2675 raft_consensus.cc:686] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 1 LEADER]: Becoming Leader. State: Replica: 68cb8bdc32d24dd5a8861ba984d80a82, State: Running, Role: LEADER
I20260926 02:56:10.108474 2675 consensus_queue.cc:237] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } } peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } }
I20260926 02:56:10.113143 2205 catalog_manager.cc:5495] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68cb8bdc32d24dd5a8861ba984d80a82 (127.19.185.194). New cstate: current_term: 1 leader_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68cb8bdc32d24dd5a8861ba984d80a82" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 41747 } health_report { overall_health: HEALTHY } } }
W20260926 02:56:10.127206 2523 tablet.cc:2369] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:56:10.128563 2391 tablet.cc:2369] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:56:10.144510 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.138946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.19.185.194:41747)
0926 02:56:10.139148 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:10.139152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:10.139170 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:56:10.139255 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:10.139259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:10.139418 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:10.139545 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:56:10.140195 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.140960 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.142359 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.142771 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.142947 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:10.143435 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:56:10.144021 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:56:10.144133 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:10.144239 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 02:56:10.144304 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:56:10.152686 2608 raft_consensus.cc:1261] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3 [term 1 FOLLOWER]: Refusing update from remote peer 68cb8bdc32d24dd5a8861ba984d80a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:56:10.152731 2345 raft_consensus.cc:1261] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20 [term 1 FOLLOWER]: Refusing update from remote peer 68cb8bdc32d24dd5a8861ba984d80a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:56:10.153702 2680 consensus_queue.cc:1035] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3661d2e3efc2472292460e1da93c8d20" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 40323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:56:10.154084 2675 consensus_queue.cc:1035] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "832b2a46504c4d509fa1f9f0bf427de3" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 39927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:56:10.171119 2684 mvcc.cc:205] Tried to move back new op lower bound from 7333443052135051264 to 7333443051967860736. Current Snapshot: MvccSnapshot[applied={T|T < 7333443052135051264}]
I20260926 02:56:10.174058 2683 mvcc.cc:205] Tried to move back new op lower bound from 7333443052135051264 to 7333443051967860736. Current Snapshot: MvccSnapshot[applied={T|T < 7333443052135051264}]
I20260926 02:56:10.179147 2204 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:56:10.179478 2204 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:56:10.183028 2204 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Sending DeleteTablet for 3 replicas of tablet 33b7c39ddc2a4ddd98829ee32a1f0a25
I20260926 02:56:10.184221 2588 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b7c39ddc2a4ddd98829ee32a1f0a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:56:10 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:35922
I20260926 02:56:10.184283 2325 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b7c39ddc2a4ddd98829ee32a1f0a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:56:10 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:33730
I20260926 02:56:10.184520 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 2242
I20260926 02:56:10.185323 2457 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b7c39ddc2a4ddd98829ee32a1f0a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:56:10 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:52618
I20260926 02:56:10.186311 2694 tablet_replica.cc:330] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82: stopping tablet replica
I20260926 02:56:10.186772 2694 raft_consensus.cc:2229] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:56:10.187549 2694 raft_consensus.cc:2258] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:56:10.198684 2694 ts_tablet_manager.cc:1905] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:56:10.210144 2693 tablet_replica.cc:330] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: stopping tablet replica
I20260926 02:56:10.210640 2693 raft_consensus.cc:2229] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:56:10.211179 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.186581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33746 (local address 127.19.185.193:40323)
0926 02:56:10.186797 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:10.186805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:10.186828 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:56:10.186896 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:10.186904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:10.187016 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:10.187142 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:10.189806 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.191331 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.193973 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.194907 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.195069 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:10.199863 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:10.199887 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:10.199911 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:10.199980 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:10.205372 (+ 5392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:10.206616 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:10.206629 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:10.206635 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:10.206855 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:10.207471 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:10.207479 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:10.207485 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:10.208176 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:10.208571 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:10.209458 (+ 887us) server_negotiation.cc:299] Negotiation successful
0926 02:56:10.209761 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:56:10.211581 2693 raft_consensus.cc:2258] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:56:10.211632 2694 ts_tablet_manager.cc:1918] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:56:10.211736 2692 tablet_replica.cc:330] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3: stopping tablet replica
I20260926 02:56:10.211917 2385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:10.211937 2694 log.cc:1196] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/33b7c39ddc2a4ddd98829ee32a1f0a25
I20260926 02:56:10.212167 2692 raft_consensus.cc:2229] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:56:10.212677 2692 raft_consensus.cc:2258] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:56:10.213199 2694 ts_tablet_manager.cc:1939] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 68cb8bdc32d24dd5a8861ba984d80a82: Deleting consensus metadata
I20260926 02:56:10.214960 2692 ts_tablet_manager.cc:1905] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:56:10.215384 2192 catalog_manager.cc:4841] TS 68cb8bdc32d24dd5a8861ba984d80a82 (127.19.185.194:41747): tablet 33b7c39ddc2a4ddd98829ee32a1f0a25 (table test-table [id=8164b82db80a46eb95ccf11796fdca67]) successfully deleted
I20260926 02:56:10.223062 2692 ts_tablet_manager.cc:1918] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:56:10.223241 2692 log.cc:1196] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/33b7c39ddc2a4ddd98829ee32a1f0a25
I20260926 02:56:10.223809 2692 ts_tablet_manager.cc:1939] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 832b2a46504c4d509fa1f9f0bf427de3: Deleting consensus metadata
I20260926 02:56:10.225781 2191 catalog_manager.cc:4841] TS 832b2a46504c4d509fa1f9f0bf427de3 (127.19.185.195:39927): tablet 33b7c39ddc2a4ddd98829ee32a1f0a25 (table test-table [id=8164b82db80a46eb95ccf11796fdca67]) successfully deleted
I20260926 02:56:10.312920 2693 ts_tablet_manager.cc:1905] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:56:10.319974 2693 ts_tablet_manager.cc:1918] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:56:10.320060 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 2242
I20260926 02:56:10.320161 2693 log.cc:1196] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/33b7c39ddc2a4ddd98829ee32a1f0a25
W20260926 02:56:10.338732 2189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.185.193:40323 (error 108)
I20260926 02:56:10.338945 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 2394
I20260926 02:56:10.340795 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.340303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40323 (local address 127.0.0.1:42836)
0926 02:56:10.340499 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:10.340619 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 02:56:10.341128 2189 catalog_manager.cc:4584] TS 3661d2e3efc2472292460e1da93c8d20 (127.19.185.193:40323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33b7c39ddc2a4ddd98829ee32a1f0a25: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40323: connect: Connection refused (error 111)
I20260926 02:56:10.357061 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.340326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.19.185.194:41747)
0926 02:56:10.340525 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:10.340530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:10.340546 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:56:10.340597 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:10.340601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:10.340676 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:10.340793 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:10.342362 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.343289 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.345258 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.345702 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.345806 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:10.349273 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:10.349313 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:10.349338 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:10.349382 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:10.353416 (+ 4034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:10.354470 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:10.354481 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:10.354485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:10.354646 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:10.355095 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:10.355100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:10.355104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:10.355705 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:10.356057 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:10.356406 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 02:56:10.356665 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 02:56:10.358016 2517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:10.394980 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.394382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40323 (local address 127.0.0.1:42848)
0926 02:56:10.394632 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:10.394789 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 02:56:10.396162 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.395724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40323 (local address 127.0.0.1:42858)
0926 02:56:10.395899 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:10.396018 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 02:56:10.456813 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.456261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40323 (local address 127.0.0.1:42862)
0926 02:56:10.456509 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:10.456622 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 02:56:10.457906 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.457525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40323 (local address 127.0.0.1:42864)
0926 02:56:10.457670 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:10.457750 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 02:56:10.491676 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 2394
I20260926 02:56:10.511339 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 2525
I20260926 02:56:10.525329 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.512095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.19.185.195:39927)
0926 02:56:10.512320 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:10.512328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:10.512407 (+ 79us) server_negotiation.cc:407] Connection header received
0926 02:56:10.512627 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:10.512631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:10.512695 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:10.512809 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:10.514277 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.514992 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.516378 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:10.516776 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:10.516912 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:10.519626 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:10.519640 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:10.519657 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:10.519699 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:10.522374 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:10.523074 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:10.523086 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:10.523091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:10.523287 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:10.523704 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:10.523710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:10.523713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:10.524080 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:10.524389 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:10.524744 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 02:56:10.524975 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:56:10.526026 2648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:10.538822 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.538142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40323 (local address 127.0.0.1:42876)
0926 02:56:10.538415 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:10.538576 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 02:56:10.540225 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:10.539651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:40323 (local address 127.0.0.1:42884)
0926 02:56:10.539942 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:10.540051 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:40323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260926 02:56:10.615130 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 2525
I20260926 02:56:10.632087 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 2172
I20260926 02:56:10.633322 2235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:10.749467 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 2172
I20260926 02:56:10.769765 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:43709
--webserver_interface=127.19.185.254
--webserver_port=43559
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:43709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:56:11.104722 2700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:11.105074 2700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:11.105275 2700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:11.112651 2700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:56:11.112771 2700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:11.112887 2700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:11.112993 2700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:56:11.113080 2700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:56:11.123275 2700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:43709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:43709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=43559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:11.124810 2700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:11.126257 2700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:11.138449 2705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:11.138384 2706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:11.138404 2708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:11.139766 2700 server_base.cc:1034] running on GCE node
I20260926 02:56:11.140652 2700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:11.142203 2700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:11.143417 2700 hybrid_clock.cc:648] HybridClock initialized: now 1790391371143352 us; error 44 us; skew 500 ppm
I20260926 02:56:11.143810 2700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:11 dist-test-slave-ktng krb5kdc[1620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391371, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:11.147339 2700 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:56:11.148924 2700 webserver.cc:460] Webserver started at http://127.19.185.254:43559/ using document root <none> and password file <none>
I20260926 02:56:11.149643 2700 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:11.149796 2700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:11.155651 2700 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:56:11.158723 2715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:11.160400 2700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:56:11.160645 2700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a"
format_stamp: "Formatted at 2026-09-26 02:56:05 on dist-test-slave-ktng"
server_key: "4bbd60f288fac3435c59598826fcb47d"
server_key_iv: "c3b7e773e4faf0e9f3af1cd5f49d3241"
server_key_version: "encryptionkey@0"
I20260926 02:56:11.161660 2700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:11.192606 2700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:11.197271 2700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:11.197559 2700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:11.217703 2700 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:43709
I20260926 02:56:11.217723 2766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:43709 every 8 connection(s)
I20260926 02:56:11.219269 2700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:56:11.223766 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 2700
I20260926 02:56:11.224641 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:40323
--local_ip_for_outbound_sockets=127.19.185.193
--tserver_master_addrs=127.19.185.254:43709
--webserver_port=42981
--webserver_interface=127.19.185.193
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:56:11.226823 2767 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:56:11.230244 2767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Bootstrap starting.
I20260926 02:56:11.250847 2767 log.cc:824] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:11.267869 2767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:56:11.268443 2767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a: Bootstrap complete.
I20260926 02:56:11.277882 2767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } }
I20260926 02:56:11.278822 2767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a7c8e8a61a4e5fb8dba29d75afdd6a, State: Initialized, Role: FOLLOWER
I20260926 02:56:11.279536 2767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } }
I20260926 02:56:11.279786 2767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:56:11.279920 2767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:56:11.280124 2767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 2 FOLLOWER]: Advancing to term 3
I20260926 02:56:11.284121 2767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } }
I20260926 02:56:11.284627 2767 leader_election.cc:302] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92a7c8e8a61a4e5fb8dba29d75afdd6a; no voters:
I20260926 02:56:11.285302 2767 leader_election.cc:288] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 02:56:11.285437 2771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 3 FOLLOWER]: Leader election won for term 3
I20260926 02:56:11.286679 2771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [term 3 LEADER]: Becoming Leader. State: Replica: 92a7c8e8a61a4e5fb8dba29d75afdd6a, State: Running, Role: LEADER
I20260926 02:56:11.287240 2771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } }
I20260926 02:56:11.287674 2767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:56:11.289785 2773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92a7c8e8a61a4e5fb8dba29d75afdd6a. Latest consensus state: current_term: 3 leader_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } } }
I20260926 02:56:11.289994 2772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a7c8e8a61a4e5fb8dba29d75afdd6a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43709 } } }
I20260926 02:56:11.290275 2773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: This master's current role is: LEADER
I20260926 02:56:11.290328 2772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a7c8e8a61a4e5fb8dba29d75afdd6a [sys.catalog]: This master's current role is: LEADER
I20260926 02:56:11.297024 2778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:56:11.305687 2778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:56:11.307179 2778 catalog_manager.cc:1259] Loaded cluster ID: 5f9696a1af4f4ecea524a6e5a0bc6a8c
I20260926 02:56:11.307284 2778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 02:56:11.312600 2778 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f27840a19ad google::LogMessage::Fail() at ??:0
@ 0x7f27840a488a google::LogMessage::SendToLog() at ??:0
@ 0x7f27840a14de google::LogMessage::Flush() at ??:0
@ 0x7f27840a2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2799ad4512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f2799ad1976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2799b59610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f2799b59439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2786ced345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2786cf091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2786cf0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2786ccbeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2789fa56db start_thread at ??:0
@ 0x7f2782f3b61f clone at ??:0
W20260926 02:56:11.618718 2769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:11.618995 2769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:11.619091 2769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:11.626506 2769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:11.626632 2769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:11.626760 2769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:56:11.636852 2769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:40323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=42981
--tserver_master_addrs=127.19.185.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:11.638145 2769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:11.639432 2769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:11.653038 2808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:11.653137 2809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:11.653138 2811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:11.654230 2769 server_base.cc:1034] running on GCE node
I20260926 02:56:11.654903 2769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:11.656070 2769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:11.657327 2769 hybrid_clock.cc:648] HybridClock initialized: now 1790391371657284 us; error 41 us; skew 500 ppm
I20260926 02:56:11.657665 2769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:11 dist-test-slave-ktng krb5kdc[1620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391371, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:11.660912 2769 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:56:11.662559 2769 webserver.cc:460] Webserver started at http://127.19.185.193:42981/ using document root <none> and password file <none>
I20260926 02:56:11.663338 2769 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:11.663506 2769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:11.669382 2769 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:56:11.674206 2818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:11.675663 2769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:56:11.675848 2769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "3661d2e3efc2472292460e1da93c8d20"
format_stamp: "Formatted at 2026-09-26 02:56:05 on dist-test-slave-ktng"
server_key: "85e73736af6d66d3a4cb7c3fe04f558a"
server_key_iv: "924aac411f45b0fb428ad414088390a3"
server_key_version: "encryptionkey@0"
I20260926 02:56:11.676828 2769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:11.699235 2769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:11.703461 2769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:11.703758 2769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:11.704759 2769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:11.706653 2825 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260926 02:56:11.712217 2825 ts_tablet_manager.cc:1852] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 02:56:11.713205 2825 ts_tablet_manager.cc:1905] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:56:11.720780 2825 ts_tablet_manager.cc:1918] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:56:11.720990 2825 log.cc:1196] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/33b7c39ddc2a4ddd98829ee32a1f0a25
I20260926 02:56:11.721503 2825 ts_tablet_manager.cc:1939] T 33b7c39ddc2a4ddd98829ee32a1f0a25 P 3661d2e3efc2472292460e1da93c8d20: Deleting consensus metadata
I20260926 02:56:11.721961 2769 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 02:56:11.722137 2769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260926 02:56:11.722318 2769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:11.722434 2769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:11.761353 2769 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:40323
I20260926 02:56:11.761404 2931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:40323 every 8 connection(s)
I20260926 02:56:11.763062 2769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:56:11.768610 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:11.766431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:52653)
0926 02:56:11.767265 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:11.767550 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":307,"threads_started":1}
W20260926 02:56:11.769363 2932 heartbeater.cc:644] Failed to heartbeat to 127.19.185.254:43709 (0 consecutive failures): Network error: Failed to ping master at 127.19.185.254:43709: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
I20260926 02:56:11.770278 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:11.769919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:48021)
0926 02:56:11.770093 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:11.770176 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 02:56:11.770498 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 2769
I20260926 02:56:11.771164 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:41747
--local_ip_for_outbound_sockets=127.19.185.194
--tserver_master_addrs=127.19.185.254:43709
--webserver_port=41233
--webserver_interface=127.19.185.194
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:56:11.771250 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:11.770870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:36769)
0926 02:56:11.771022 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:11.771109 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 02:56:11.771565 2932 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:56:12.106979 2935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:12.107268 2935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:12.107471 2935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:12.115130 2935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:12.115247 2935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:12.115459 2935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:56:12.125607 2935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:41747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=41233
--tserver_master_addrs=127.19.185.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:12.126983 2935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:12.128464 2935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:12.141014 2940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:12.141950 2941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:12.142257 2943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:12.143361 2935 server_base.cc:1034] running on GCE node
I20260926 02:56:12.144030 2935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:12.145036 2935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:12.146284 2935 hybrid_clock.cc:648] HybridClock initialized: now 1790391372146241 us; error 67 us; skew 500 ppm
I20260926 02:56:12.146641 2935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:12 dist-test-slave-ktng krb5kdc[1620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391372, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:12.149883 2935 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:56:12.151394 2935 webserver.cc:460] Webserver started at http://127.19.185.194:41233/ using document root <none> and password file <none>
I20260926 02:56:12.152117 2935 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:12.152264 2935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:12.157581 2935 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 02:56:12.160382 2950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:12.162467 2935 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 02:56:12.162635 2935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "68cb8bdc32d24dd5a8861ba984d80a82"
format_stamp: "Formatted at 2026-09-26 02:56:06 on dist-test-slave-ktng"
server_key: "4c2df08625b0925a02694fd46a7beff1"
server_key_iv: "a638b99b7617c4e5d0377951ca017e80"
server_key_version: "encryptionkey@0"
I20260926 02:56:12.164129 2935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:12.197633 2935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:12.202118 2935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:12.202402 2935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:12.203470 2935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:12.204865 2935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:56:12.204993 2935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:12.205124 2935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:12.205212 2935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:12.239866 2935 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:41747
I20260926 02:56:12.239954 3062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:41747 every 8 connection(s)
I20260926 02:56:12.241364 2935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:56:12.245800 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 2935
I20260926 02:56:12.246415 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:39927
--local_ip_for_outbound_sockets=127.19.185.195
--tserver_master_addrs=127.19.185.254:43709
--webserver_port=34853
--webserver_interface=127.19.185.195
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:56:12.246429 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:12.244511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:53737)
0926 02:56:12.245234 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:12.245471 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":279,"threads_started":1}
W20260926 02:56:12.247164 3063 heartbeater.cc:644] Failed to heartbeat to 127.19.185.254:43709 (0 consecutive failures): Network error: Failed to ping master at 127.19.185.254:43709: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
I20260926 02:56:12.248168 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:12.247704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:46383)
0926 02:56:12.247904 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:12.247998 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 02:56:12.249302 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:12.248867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:35525)
0926 02:56:12.249044 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:12.249144 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 02:56:12.249608 3063 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:56:12.577642 3066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:12.577944 3066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:12.578107 3066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:12.585448 3066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:12.585564 3066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:12.585740 3066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:56:12.595777 3066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:39927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=34853
--tserver_master_addrs=127.19.185.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:12.597097 3066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:12.598417 3066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:12.611497 3071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:12.611457 3072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:12.612861 3074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:12.614194 3066 server_base.cc:1034] running on GCE node
I20260926 02:56:12.615036 3066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:12.616182 3066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:12.617432 3066 hybrid_clock.cc:648] HybridClock initialized: now 1790391372617373 us; error 51 us; skew 500 ppm
I20260926 02:56:12.617774 3066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:12 dist-test-slave-ktng krb5kdc[1620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391372, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:12.621243 3066 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:56:12.622787 3066 webserver.cc:460] Webserver started at http://127.19.185.195:34853/ using document root <none> and password file <none>
I20260926 02:56:12.623641 3066 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:12.623811 3066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:12.629480 3066 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:56:12.632311 3081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:12.633723 3066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:56:12.633915 3066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "832b2a46504c4d509fa1f9f0bf427de3"
format_stamp: "Formatted at 2026-09-26 02:56:06 on dist-test-slave-ktng"
server_key: "faceeda24c63262457856d6d74bd9585"
server_key_iv: "40fa638b596c4c444f8799745dac8799"
server_key_version: "encryptionkey@0"
I20260926 02:56:12.634917 3066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:12.663671 3066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:12.668201 3066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:12.668488 3066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:12.669590 3066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:12.671022 3066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:56:12.671139 3066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:12.671275 3066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:12.671387 3066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 02:56:12.705590 3066 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:39927
I20260926 02:56:12.705675 3193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:39927 every 8 connection(s)
I20260926 02:56:12.707075 3066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:56:12.710354 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 3066
I20260926 02:56:12.712723 3196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:12.710412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:35587)
0926 02:56:12.711362 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:12.711657 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":354,"threads_started":1}
W20260926 02:56:12.713534 3194 heartbeater.cc:644] Failed to heartbeat to 127.19.185.254:43709 (0 consecutive failures): Network error: Failed to ping master at 127.19.185.254:43709: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
I20260926 02:56:12.714336 3196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:12.714027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:58835)
0926 02:56:12.714166 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:12.714234 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 02:56:12.715205 3196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:12.714841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:58895)
0926 02:56:12.715031 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:12.715103 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 02:56:12.715425 3194 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 02:56:12.773034 3198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:12.772201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:50017)
0926 02:56:12.772688 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:12.772782 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":243,"threads_started":1}
I20260926 02:56:13.251116 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:13.250177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:34831)
0926 02:56:13.250753 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:13.250891 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":288,"threads_started":1}
I20260926 02:56:13.716727 3200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:13.715941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:56675)
0926 02:56:13.716378 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:13.716508 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":205,"threads_started":1}
I20260926 02:56:13.774780 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:13.774110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:52027)
0926 02:56:13.774541 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:13.774627 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":231,"threads_started":1}
I20260926 02:56:14.252766 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:14.252040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:56871)
0926 02:56:14.252516 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:14.252599 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":245,"threads_started":1}
I20260926 02:56:14.718333 3203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:14.717601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:41151)
0926 02:56:14.718075 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:14.718157 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":240,"threads_started":1}
I20260926 02:56:14.776738 3204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:14.775836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:48721)
0926 02:56:14.776420 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:14.776549 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":300,"threads_started":1}
I20260926 02:56:15.254446 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:15.253715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:57593)
0926 02:56:15.254170 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:15.254256 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":244,"threads_started":1}
I20260926 02:56:15.720038 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:15.719217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:33041)
0926 02:56:15.719783 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:15.719862 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":382,"threads_started":1}
I20260926 02:56:15.778475 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:15.777689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:49367)
0926 02:56:15.778230 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:15.778312 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":330,"threads_started":1}
I20260926 02:56:16.256166 3208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:16.255426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:57709)
0926 02:56:16.255906 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:16.255992 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":243,"threads_started":1}
I20260926 02:56:16.721683 3209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:16.720950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:35815)
0926 02:56:16.721430 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:16.721513 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":254,"threads_started":1}
I20260926 02:56:16.780228 3210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:16.779407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:41679)
0926 02:56:16.779971 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:16.780052 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":316,"threads_started":1}
I20260926 02:56:17.257902 3211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:17.257128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:39881)
0926 02:56:17.257624 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:17.257706 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":247,"threads_started":1}
I20260926 02:56:17.723599 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:17.722721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:39491)
0926 02:56:17.723288 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:17.723426 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":263,"threads_started":1}
I20260926 02:56:17.781932 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:17.781186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:59539)
0926 02:56:17.781681 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:17.781763 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":284,"threads_started":1}
I20260926 02:56:18.259897 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:18.259066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:34255)
0926 02:56:18.259611 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:18.259711 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":317,"threads_started":1}
I20260926 02:56:18.725411 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:18.724592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:51339)
0926 02:56:18.725144 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:18.725237 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":315,"threads_started":1}
I20260926 02:56:18.783792 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:18.782868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:46477)
0926 02:56:18.783479 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:18.783577 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":246,"threads_started":1}
I20260926 02:56:19.261770 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:19.260896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:45505)
0926 02:56:19.261409 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:19.261555 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":209,"threads_started":1}
I20260926 02:56:19.727514 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:19.726544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:40009)
0926 02:56:19.727145 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:19.727245 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":305,"threads_started":1}
I20260926 02:56:19.785681 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:19.784902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:33015)
0926 02:56:19.785416 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:19.785501 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":267,"threads_started":1}
I20260926 02:56:20.263641 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:20.262720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:45997)
0926 02:56:20.263325 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:20.263444 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":289,"threads_started":1}
I20260926 02:56:20.729462 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:20.728665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:51415)
0926 02:56:20.729180 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:20.729293 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":284,"threads_started":1}
I20260926 02:56:20.787624 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:20.786719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:35349)
0926 02:56:20.787259 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:20.787440 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":298,"threads_started":1}
I20260926 02:56:21.265489 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:21.264690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:56867)
0926 02:56:21.265204 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:21.265291 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":259,"threads_started":1}
I20260926 02:56:21.731410 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:21.730517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:52193)
0926 02:56:21.731066 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:21.731152 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":258,"threads_started":1}
I20260926 02:56:21.789439 3225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:21.788731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:35677)
0926 02:56:21.789177 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:21.789261 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":240,"threads_started":1}
I20260926 02:56:22.267592 3226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:22.266626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:50905)
0926 02:56:22.267221 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:22.267389 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":338,"threads_started":1}
I20260926 02:56:22.733099 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:22.732394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:59219)
0926 02:56:22.732838 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:22.732933 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":245,"threads_started":1}
I20260926 02:56:22.791112 3228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:22.790360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:53621)
0926 02:56:22.790819 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:22.790909 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":259,"threads_started":1}
I20260926 02:56:23.269366 3229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:23.268652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:52779)
0926 02:56:23.269087 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:23.269183 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":218,"threads_started":1}
I20260926 02:56:23.734812 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:23.734013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:34217)
0926 02:56:23.734541 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:23.734637 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":282,"threads_started":1}
I20260926 02:56:23.792800 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:23.792088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:48889)
0926 02:56:23.792544 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:23.792630 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":247,"threads_started":1}
I20260926 02:56:24.271802 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:24.270445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:58377)
0926 02:56:24.271549 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:24.271639 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":790,"threads_started":1}
I20260926 02:56:24.736509 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:24.735833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:55809)
0926 02:56:24.736242 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:24.736325 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":206,"threads_started":1}
I20260926 02:56:24.794750 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:24.793856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:59945)
0926 02:56:24.794386 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:24.794537 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":301,"threads_started":1}
I20260926 02:56:25.273630 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:25.272828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:46407)
0926 02:56:25.273333 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:25.273433 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":257,"threads_started":1}
I20260926 02:56:25.738546 3236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:25.737510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:41331)
0926 02:56:25.738169 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:25.738263 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":441,"threads_started":1}
I20260926 02:56:25.796505 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:25.795723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:47217)
0926 02:56:25.796184 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:25.796271 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":229,"threads_started":1}
I20260926 02:56:26.275279 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:26.274552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:56095)
0926 02:56:26.275009 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:26.275095 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":232,"threads_started":1}
I20260926 02:56:26.740209 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:26.739475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:42991)
0926 02:56:26.739954 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:26.740033 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":257,"threads_started":1}
I20260926 02:56:26.798662 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:26.797810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.193:32975)
0926 02:56:26.798350 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:26.798477 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":259,"threads_started":1}
I20260926 02:56:27.277567 3241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:27.276601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.194:58301)
0926 02:56:27.277258 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:27.277380 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":427,"threads_started":1}
I20260926 02:56:27.721194 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 2769
I20260926 02:56:27.742297 3243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:27.741401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:43709 (local address 127.19.185.195:48681)
0926 02:56:27.742001 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:27.742122 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
I20260926 02:56:27.746757 3242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:27.722587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.19.185.193:40323)
0926 02:56:27.723364 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:27.723386 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:27.723446 (+ 60us) server_negotiation.cc:407] Connection header received
0926 02:56:27.725271 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:27.725283 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:27.725753 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:27.726073 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:27.727721 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:27.729025 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:27.730368 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:27.731081 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:27.731224 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:27.739749 (+ 8525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:27.739767 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:27.739787 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:27.739844 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:27.743504 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:27.744286 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:27.744294 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:27.744301 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:27.744491 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:27.744818 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:27.744821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:27.744823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:27.745400 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:27.745656 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:27.746017 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 02:56:27.746470 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":382,"threads_started":1}
I20260926 02:56:27.748363 2927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:27.834503 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 2769
I20260926 02:56:27.852550 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 2935
I20260926 02:56:27.870545 3246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:27.853623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.19.185.194:41747)
0926 02:56:27.854189 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:27.854203 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:27.854240 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:56:27.855859 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:27.855869 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:27.856304 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:27.856612 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:27.857937 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:27.859186 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:27.860596 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:27.861308 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:27.861477 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:27.864215 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:27.864223 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:27.864240 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:27.864277 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:27.867109 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:27.867847 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:27.867855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:27.867863 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:27.868089 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:27.868439 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:27.868442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:27.868445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:27.869009 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:27.869255 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:27.869689 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 02:56:27.870152 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":305,"threads_started":1}
I20260926 02:56:27.872097 3058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:27.958251 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 2935
I20260926 02:56:27.976555 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 3066
I20260926 02:56:27.995647 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:27.977594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.19.185.195:39927)
0926 02:56:27.978201 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:27.978215 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:27.978256 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:56:27.979978 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:27.979988 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:27.980488 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:27.980858 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:27.982452 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:27.983856 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:27.985228 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:27.985904 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:27.986107 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:27.989010 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:27.989017 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:27.989032 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:27.989080 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:27.992127 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:27.992921 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:27.992929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:27.992935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:27.993098 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:27.993477 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:27.993484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:27.993487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:27.994057 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:27.994371 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:27.994811 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 02:56:27.995245 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":281,"threads_started":1}
I20260926 02:56:27.997227 3189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:28.081784 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 3066
2026-09-26T02:56:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24523 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:56:28 dist-test-slave-ktng krb5kdc[3252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:56:28 dist-test-slave-ktng krb5kdc[3252](info): set up 2 sockets
Sep 26 02:56:28 dist-test-slave-ktng krb5kdc[3252](info): commencing operation
krb5kdc: starting...
W20260926 02:56:29.276799 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:56:29 dist-test-slave-ktng krb5kdc[3252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:56:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:56:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:56:29.544344 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:38225
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:38225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:56:29.874714 3265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:29.875008 3265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:29.875151 3265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:29.882444 3265 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 02:56:29.882550 3265 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 02:56:29.882623 3265 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 02:56:29.882700 3265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:56:29.882795 3265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:29.882927 3265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:29.883180 3265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:56:29.883353 3265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:56:29.893720 3265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:38225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:38225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:29.895082 3265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:29.896548 3265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:29.908847 3271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:29.908896 3273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:29.909067 3270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:29.909557 3265 server_base.cc:1034] running on GCE node
I20260926 02:56:29.910305 3265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:29.911798 3265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:29.913028 3265 hybrid_clock.cc:648] HybridClock initialized: now 1790391389912948 us; error 50 us; skew 500 ppm
I20260926 02:56:29.913390 3265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:29 dist-test-slave-ktng krb5kdc[3252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391389, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:29.916754 3265 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:56:29.918253 3265 webserver.cc:460] Webserver started at http://127.19.185.254:38933/ using document root <none> and password file <none>
I20260926 02:56:29.918953 3265 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:29.919086 3265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:29.919489 3265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:56:29.922479 3265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "c15a442b8f874e159320957ea107c032"
format_stamp: "Formatted at 2026-09-26 02:56:29 on dist-test-slave-ktng"
server_key: "3086d6661544b8fdea6ddb41926597c4"
server_key_iv: "39258ae319e978679bec1f10d38e4726"
server_key_version: "encryptionkey@0"
I20260926 02:56:29.923271 3265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "c15a442b8f874e159320957ea107c032"
format_stamp: "Formatted at 2026-09-26 02:56:29 on dist-test-slave-ktng"
server_key: "3086d6661544b8fdea6ddb41926597c4"
server_key_iv: "39258ae319e978679bec1f10d38e4726"
server_key_version: "encryptionkey@0"
I20260926 02:56:29.928622 3265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 02:56:29.932087 3280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:29.933530 3265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:56:29.933729 3265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "c15a442b8f874e159320957ea107c032"
format_stamp: "Formatted at 2026-09-26 02:56:29 on dist-test-slave-ktng"
server_key: "3086d6661544b8fdea6ddb41926597c4"
server_key_iv: "39258ae319e978679bec1f10d38e4726"
server_key_version: "encryptionkey@0"
I20260926 02:56:29.933950 3265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:29.949819 3265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:29.954344 3265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:29.954620 3265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:29.974082 3265 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:38225
I20260926 02:56:29.974112 3331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:38225 every 8 connection(s)
I20260926 02:56:29.975574 3265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:56:29.977517 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 3265
I20260926 02:56:29.977764 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:56:29.978477 20199 external_mini_cluster.cc:1428] Setting key 1aacfc4c3f6e92d7c047f16bb84fbdee
I20260926 02:56:29.979779 3332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:56:29 dist-test-slave-ktng krb5kdc[3252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:29.994441 3332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Bootstrap starting.
I20260926 02:56:29.997874 3332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:29.998908 3332 log.cc:824] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:30.001858 3332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: No bootstrap required, opened a new log
I20260926 02:56:30.002005 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:29.980520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.19.185.254:38225)
0926 02:56:29.981323 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:29.981335 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:29.981373 (+ 38us) server_negotiation.cc:407] Connection header received
0926 02:56:29.982877 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:29.982897 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:29.983272 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:29.983670 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:29.985538 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:29.986917 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:29.988819 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:29.989527 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:29.989660 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:29.994637 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:29.994653 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:29.994670 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:29.994715 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:29.997846 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:29.998608 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:29.998620 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:29.998626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:29.998801 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:29.999237 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:29.999243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:29.999246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:29.999848 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:30.000175 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:30.000607 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 02:56:30.001043 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":361,"threads_started":1}
I20260926 02:56:30.009729 3332 raft_consensus.cc:348] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } }
I20260926 02:56:30.010155 3332 raft_consensus.cc:374] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:30.010272 3332 raft_consensus.cc:729] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15a442b8f874e159320957ea107c032, State: Initialized, Role: FOLLOWER
I20260926 02:56:30.010900 3332 consensus_queue.cc:260] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } }
I20260926 02:56:30.011147 3332 raft_consensus.cc:388] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:56:30.011284 3332 raft_consensus.cc:482] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:56:30.011518 3332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:56:30.013649 3332 raft_consensus.cc:504] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } }
I20260926 02:56:30.014333 3332 leader_election.cc:302] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c15a442b8f874e159320957ea107c032; no voters:
I20260926 02:56:30.014941 3332 leader_election.cc:288] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:56:30.015105 3337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:56:30.015827 3337 raft_consensus.cc:686] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 1 LEADER]: Becoming Leader. State: Replica: c15a442b8f874e159320957ea107c032, State: Running, Role: LEADER
I20260926 02:56:30.016355 3337 consensus_queue.cc:237] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } }
I20260926 02:56:30.017097 3332 sys_catalog.cc:564] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:56:30.018939 3339 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c15a442b8f874e159320957ea107c032. Latest consensus state: current_term: 1 leader_uuid: "c15a442b8f874e159320957ea107c032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } } }
I20260926 02:56:30.019321 3339 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: This master's current role is: LEADER
I20260926 02:56:30.020052 3338 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c15a442b8f874e159320957ea107c032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } } }
I20260926 02:56:30.020308 3342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:56:30.020402 3338 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: This master's current role is: LEADER
I20260926 02:56:30.026196 3342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:56:30.034857 3342 catalog_manager.cc:1347] Generated new cluster ID: b1e8173b89804c2fa71a9bed1486c255
I20260926 02:56:30.034976 3342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:56:30.055238 3342 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:56:30.056725 3342 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:56:30.066504 3342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Generated new TSK 0
I20260926 02:56:30.067459 3342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:56:30.139808 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38225
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:56:30.475843 3358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:30.476130 3358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:30.476230 3358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:30.484107 3358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:30.484213 3358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:30.484328 3358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:56:30.494110 3358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:30.495442 3358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:30.496749 3358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:30.511165 3364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:30.511193 3363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:30.511941 3366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:30.512053 3358 server_base.cc:1034] running on GCE node
I20260926 02:56:30.512696 3358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:30.513661 3358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:30.514895 3358 hybrid_clock.cc:648] HybridClock initialized: now 1790391390514838 us; error 43 us; skew 500 ppm
I20260926 02:56:30.515187 3358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:30 dist-test-slave-ktng krb5kdc[3252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:30.518790 3358 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:56:30.520370 3358 webserver.cc:460] Webserver started at http://127.19.185.193:43913/ using document root <none> and password file <none>
I20260926 02:56:30.521073 3358 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:30.521219 3358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:30.521538 3358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:56:30.524753 3358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "6b1ff144506a47f0876a360d9728c818"
format_stamp: "Formatted at 2026-09-26 02:56:30 on dist-test-slave-ktng"
server_key: "2dbe18d8c82c8ac975c8933c5128afce"
server_key_iv: "5e0791b026124e75cec065f8920ffb2a"
server_key_version: "encryptionkey@0"
I20260926 02:56:30.525501 3358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "6b1ff144506a47f0876a360d9728c818"
format_stamp: "Formatted at 2026-09-26 02:56:30 on dist-test-slave-ktng"
server_key: "2dbe18d8c82c8ac975c8933c5128afce"
server_key_iv: "5e0791b026124e75cec065f8920ffb2a"
server_key_version: "encryptionkey@0"
I20260926 02:56:30.530741 3358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:56:30.534219 3373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:30.535797 3358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:56:30.535974 3358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "6b1ff144506a47f0876a360d9728c818"
format_stamp: "Formatted at 2026-09-26 02:56:30 on dist-test-slave-ktng"
server_key: "2dbe18d8c82c8ac975c8933c5128afce"
server_key_iv: "5e0791b026124e75cec065f8920ffb2a"
server_key_version: "encryptionkey@0"
I20260926 02:56:30.536173 3358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:30.548359 3358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:30.552618 3358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:30.553012 3358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:30.553965 3358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:30.555334 3358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:56:30.555438 3358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:30.555593 3358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:30.555666 3358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:30.590833 3358 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:46763
I20260926 02:56:30.590883 3485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:46763 every 8 connection(s)
I20260926 02:56:30.592237 3358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:56:30.594010 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 3358
I20260926 02:56:30.594236 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:56:30.595096 20199 external_mini_cluster.cc:1428] Setting key 079432f2e206a0e35fe2b9167b0285e4
Sep 26 02:56:30 dist-test-slave-ktng krb5kdc[3252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:30.614405 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:30.594990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:42565 (local address 127.19.185.254:38225)
0926 02:56:30.595461 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:30.595466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:30.596837 (+ 1371us) server_negotiation.cc:407] Connection header received
0926 02:56:30.598734 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:30.598744 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:30.598844 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:30.598977 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:30.600567 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:30.601721 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:30.603946 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:30.604670 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:30.604820 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:30.608586 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:30.608591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:30.608594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:30.608628 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:30.611214 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:30.611993 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:30.611998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:30.612006 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:30.612126 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:30.612588 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:30.612593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:30.612596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:30.612888 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:30.613043 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:30.613775 (+ 732us) server_negotiation.cc:299] Negotiation successful
0926 02:56:30.614029 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":229,"threads_started":1}
I20260926 02:56:30.615530 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:30.595775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:42565)
0926 02:56:30.596664 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:30.596717 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:30.598454 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:30.599166 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:30.599174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:30.599817 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:56:30.600389 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:30.600408 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:30.601892 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:30.601896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:30.603787 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:30.603806 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:30.604942 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:30.604948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:30.605165 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:30.606027 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:56:30.606049 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:56:30.608334 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:56:30.611388 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:30.611398 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:30.611415 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:30.611848 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:30.612240 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:30.612245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:30.612249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:30.612445 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:30.613028 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:56:30.613036 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:56:30.613378 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 02:56:30.613660 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 02:56:30.614224 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":346,"threads_started":1}
I20260926 02:56:30.617249 3486 heartbeater.cc:347] Connected to a master server at 127.19.185.254:38225
I20260926 02:56:30.617587 3486 heartbeater.cc:464] Registering TS with master...
I20260926 02:56:30.618382 3486 heartbeater.cc:511] Master 127.19.185.254:38225 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
I20260926 02:56:30.620653 3297 ts_manager.cc:194] Registered new tserver with Master: 6b1ff144506a47f0876a360d9728c818 (127.19.185.193:46763)
I20260926 02:56:30.622228 3297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:42565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:56:30.650960 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38225
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:56:30.987488 3492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:30.987820 3492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:30.988003 3492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:30.995620 3492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:30.995769 3492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:30.995967 3492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:56:31.006278 3492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:31.007706 3492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:31.008993 3492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:31.025306 3498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:31.025832 3497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:31.026095 3500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:31.026429 3492 server_base.cc:1034] running on GCE node
I20260926 02:56:31.027109 3492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:31.028224 3492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:31.029460 3492 hybrid_clock.cc:648] HybridClock initialized: now 1790391391029403 us; error 54 us; skew 500 ppm
I20260926 02:56:31.029780 3492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:31 dist-test-slave-ktng krb5kdc[3252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391391, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:31.033437 3492 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:56:31.035130 3492 webserver.cc:460] Webserver started at http://127.19.185.194:41671/ using document root <none> and password file <none>
I20260926 02:56:31.035900 3492 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:31.036056 3492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:31.036386 3492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:56:31.039860 3492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "94d5768070d84037a35f0ba378c5e3f8"
format_stamp: "Formatted at 2026-09-26 02:56:31 on dist-test-slave-ktng"
server_key: "e68ca22310fe5fbf5278953256ca82ed"
server_key_iv: "c436552ef6ea009dd05a32ddea31f919"
server_key_version: "encryptionkey@0"
I20260926 02:56:31.040652 3492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "94d5768070d84037a35f0ba378c5e3f8"
format_stamp: "Formatted at 2026-09-26 02:56:31 on dist-test-slave-ktng"
server_key: "e68ca22310fe5fbf5278953256ca82ed"
server_key_iv: "c436552ef6ea009dd05a32ddea31f919"
server_key_version: "encryptionkey@0"
I20260926 02:56:31.046559 3492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:56:31.050711 3507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:31.052439 3492 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:56:31.052647 3492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "94d5768070d84037a35f0ba378c5e3f8"
format_stamp: "Formatted at 2026-09-26 02:56:31 on dist-test-slave-ktng"
server_key: "e68ca22310fe5fbf5278953256ca82ed"
server_key_iv: "c436552ef6ea009dd05a32ddea31f919"
server_key_version: "encryptionkey@0"
I20260926 02:56:31.052879 3492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:31.082406 3492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:31.086953 3492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:31.087263 3492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:31.088387 3492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:31.089883 3492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:56:31.089984 3492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:31.090140 3492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:31.090215 3492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:31.132869 3492 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:34651
I20260926 02:56:31.132962 3619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:34651 every 8 connection(s)
I20260926 02:56:31.134181 3492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:56:31.139086 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 3492
I20260926 02:56:31.139271 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:56:31.139725 20199 external_mini_cluster.cc:1428] Setting key cca688093ad475957852bf187ce0a8c7
Sep 26 02:56:31 dist-test-slave-ktng krb5kdc[3252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391391, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:31.157337 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.136918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:59599 (local address 127.19.185.254:38225)
0926 02:56:31.137397 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:31.137403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:31.138523 (+ 1120us) server_negotiation.cc:407] Connection header received
0926 02:56:31.140791 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:31.140795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:31.140864 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:31.141082 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:31.143106 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.143865 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.146371 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.146822 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.146938 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.151206 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:31.151210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:31.151214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:31.151248 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:31.154057 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:31.154812 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:31.154824 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:31.154827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:31.154958 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:31.155438 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:31.155444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:31.155447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:31.155721 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:31.155886 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:31.156668 (+ 782us) server_negotiation.cc:299] Negotiation successful
0926 02:56:31.156930 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":221,"threads_started":1}
I20260926 02:56:31.158538 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.137573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:59599)
0926 02:56:31.138363 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:31.138411 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:31.140284 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:31.141363 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:31.141374 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:31.142114 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:56:31.142926 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:31.142954 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.144034 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.144042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:31.146197 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:31.146215 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.147063 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.147072 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:31.147468 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.148234 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:56:31.148269 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:56:31.150957 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:56:31.154212 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:31.154221 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:31.154239 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:31.154680 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:31.155073 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:31.155078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:31.155081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:31.155276 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:31.155859 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:56:31.155867 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:56:31.156279 (+ 412us) client_negotiation.cc:769] Sending connection context
0926 02:56:31.156554 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 02:56:31.157245 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":320,"threads_started":1}
I20260926 02:56:31.160079 3620 heartbeater.cc:347] Connected to a master server at 127.19.185.254:38225
I20260926 02:56:31.160425 3620 heartbeater.cc:464] Registering TS with master...
I20260926 02:56:31.161392 3620 heartbeater.cc:511] Master 127.19.185.254:38225 requested a full tablet report, sending...
I20260926 02:56:31.163259 3297 ts_manager.cc:194] Registered new tserver with Master: 94d5768070d84037a35f0ba378c5e3f8 (127.19.185.194:34651)
I20260926 02:56:31.164595 3297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:59599
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:56:31.198334 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38225
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:56:31.524633 3626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:31.524917 3626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:31.525018 3626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:31.532186 3626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:31.532289 3626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:31.532400 3626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:56:31.542094 3626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:31.543355 3626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:31.544595 3626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:31.557477 3631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:31.557508 3634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:31.557472 3632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:31.558413 3626 server_base.cc:1034] running on GCE node
I20260926 02:56:31.559052 3626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:31.560175 3626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:31.561408 3626 hybrid_clock.cc:648] HybridClock initialized: now 1790391391561332 us; error 61 us; skew 500 ppm
I20260926 02:56:31.561721 3626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:31 dist-test-slave-ktng krb5kdc[3252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391391, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:31.564795 3626 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:56:31.566179 3626 webserver.cc:460] Webserver started at http://127.19.185.195:32891/ using document root <none> and password file <none>
I20260926 02:56:31.566854 3626 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:31.567008 3626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:31.567363 3626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:56:31.570257 3626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "9c26acfcdf844eb7b8476d1aa2018f76"
format_stamp: "Formatted at 2026-09-26 02:56:31 on dist-test-slave-ktng"
server_key: "60d0ab52cafc7c2b8bf33cc1d391ecf5"
server_key_iv: "565747288b75dac7d8801f55ec9728e3"
server_key_version: "encryptionkey@0"
I20260926 02:56:31.571075 3626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "9c26acfcdf844eb7b8476d1aa2018f76"
format_stamp: "Formatted at 2026-09-26 02:56:31 on dist-test-slave-ktng"
server_key: "60d0ab52cafc7c2b8bf33cc1d391ecf5"
server_key_iv: "565747288b75dac7d8801f55ec9728e3"
server_key_version: "encryptionkey@0"
I20260926 02:56:31.576349 3626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:56:31.579841 3641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:31.581182 3626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 02:56:31.581354 3626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "9c26acfcdf844eb7b8476d1aa2018f76"
format_stamp: "Formatted at 2026-09-26 02:56:31 on dist-test-slave-ktng"
server_key: "60d0ab52cafc7c2b8bf33cc1d391ecf5"
server_key_iv: "565747288b75dac7d8801f55ec9728e3"
server_key_version: "encryptionkey@0"
I20260926 02:56:31.581557 3626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:31.609332 3626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:31.613493 3626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:31.613811 3626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:31.614804 3626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:31.616248 3626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:56:31.616351 3626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:31.616505 3626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:31.616581 3626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:31.625675 3486 heartbeater.cc:503] Master 127.19.185.254:38225 was elected leader, sending a full tablet report...
I20260926 02:56:31.654184 3626 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:35819
I20260926 02:56:31.654230 3753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:35819 every 8 connection(s)
I20260926 02:56:31.655591 3626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:56:31.662367 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 3626
I20260926 02:56:31.662561 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:56:31.663007 20199 external_mini_cluster.cc:1428] Setting key 4afa8178e0d65601a1d916ebf9bbc6df
Sep 26 02:56:31 dist-test-slave-ktng krb5kdc[3252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391391, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:31.676822 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.658621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:58249 (local address 127.19.185.254:38225)
0926 02:56:31.659088 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:31.659092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:31.660034 (+ 942us) server_negotiation.cc:407] Connection header received
0926 02:56:31.661885 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:31.661890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:31.661997 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:31.662118 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:31.663972 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.665088 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.666776 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.667193 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.667352 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.670752 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:31.670758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:31.670763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:31.670805 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:31.673677 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:31.674413 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:31.674426 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:31.674429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:31.674551 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:31.674981 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:31.674986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:31.674989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:31.675274 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:31.675524 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:31.676214 (+ 690us) server_negotiation.cc:299] Negotiation successful
0926 02:56:31.676444 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":248,"threads_started":1}
I20260926 02:56:31.677850 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.659075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:58249)
0926 02:56:31.659864 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:31.659905 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:31.661608 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:31.662365 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:31.662374 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:31.662996 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:56:31.663769 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:31.663797 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.665226 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.665232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:31.666619 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:31.666631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.667452 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.667459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:31.667680 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.668370 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:56:31.668391 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:56:31.670545 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:56:31.673825 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:31.673834 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:31.673852 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:31.674279 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:31.674662 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:31.674667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:31.674671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:31.674854 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:31.675483 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:56:31.675492 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:56:31.675827 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 02:56:31.676108 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 02:56:31.676629 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":372,"threads_started":1}
I20260926 02:56:31.679400 3754 heartbeater.cc:347] Connected to a master server at 127.19.185.254:38225
I20260926 02:56:31.679721 3754 heartbeater.cc:464] Registering TS with master...
I20260926 02:56:31.680469 3754 heartbeater.cc:511] Master 127.19.185.254:38225 requested a full tablet report, sending...
I20260926 02:56:31.681965 3297 ts_manager.cc:194] Registered new tserver with Master: 9c26acfcdf844eb7b8476d1aa2018f76 (127.19.185.195:35819)
I20260926 02:56:31.682778 3297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:58249
I20260926 02:56:31.690940 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:56:31.708832 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.695788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.19.185.254:38225)
0926 02:56:31.695961 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:31.695965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:31.696414 (+ 449us) server_negotiation.cc:407] Connection header received
0926 02:56:31.696548 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:31.696551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:31.696627 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:31.696706 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:31.698111 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.698793 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.700275 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.700649 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.700740 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.703440 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:31.703450 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:31.703457 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:31.703506 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:31.706017 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:31.706681 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:31.706690 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:31.706693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:31.706823 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:31.707216 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:31.707221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:31.707225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:31.707634 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:31.707888 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:31.708265 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 02:56:31.708503 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:56:31.712930 3297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:56:31.719914 3297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:56:31.742729 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.732576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.19.185.194:34651)
0926 02:56:31.733387 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:31.733399 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:31.733835 (+ 436us) server_negotiation.cc:407] Connection header received
0926 02:56:31.734242 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:31.734252 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:31.734498 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:31.734673 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:31.735835 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.737355 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.740058 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.741548 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.741734 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.741926 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:31.742224 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 02:56:31.742393 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":416,"threads_started":1}
I20260926 02:56:31.742801 3769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.732319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:34651 (local address 127.0.0.1:45952)
0926 02:56:31.733690 (+ 1371us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:31.733737 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:31.734006 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:31.735043 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:31.735049 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:31.735082 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:31.735589 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:31.735604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.737506 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.737512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:31.739830 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:31.739845 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.741771 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.741777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:31.741987 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.742040 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 02:56:31.742182 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 02:56:31.742375 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"mutex_wait_us":67,"spinlock_wait_cycles":11776,"thread_start_us":1263,"threads_started":1}
I20260926 02:56:31.743866 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.732678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46763 (local address 127.0.0.1:53478)
0926 02:56:31.733831 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:31.733848 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:31.734005 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:31.735234 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:31.735245 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:31.735271 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:31.735880 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:31.735897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.738090 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.738096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:31.740290 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:31.740305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.743052 (+ 2747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.743057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:31.743174 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.743196 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 02:56:31.743305 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 02:56:31.743453 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":23,"spinlock_wait_cycles":23296,"thread_start_us":818,"threads_started":1}
I20260926 02:56:31.743880 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.733295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.19.185.193:46763)
0926 02:56:31.734613 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:31.734622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:31.734651 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:56:31.734746 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:31.734754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:31.734993 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:31.735874 (+ 881us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:31.736104 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.737901 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.740452 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.742057 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.742209 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.742377 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:31.743381 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0926 02:56:31.743538 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1126,"thread_start_us":1561,"threads_started":1}
I20260926 02:56:31.744693 3771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.734328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:35819 (local address 127.0.0.1:46900)
0926 02:56:31.736272 (+ 1944us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:31.736292 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:31.736425 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:31.736894 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:31.736898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:31.736921 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:31.737400 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:31.737413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.739538 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.739544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:31.741774 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:31.741791 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.743964 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.743970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:31.744126 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.744158 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:56:31.744250 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 02:56:31.744374 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1852,"thread_start_us":1704,"threads_started":1}
I20260926 02:56:31.744742 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.734838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46900 (local address 127.19.185.195:35819)
0926 02:56:31.735590 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:31.735598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:31.736387 (+ 789us) server_negotiation.cc:407] Connection header received
0926 02:56:31.736588 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:31.736592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:31.736730 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:31.736904 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:31.737623 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.739387 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.741967 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.743479 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.743656 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.743816 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:31.744315 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 02:56:31.744436 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":390,"threads_started":1}
I20260926 02:56:31.747046 3555 tablet_service.cc:1467] Processing CreateTablet for tablet b74b4aba12c0482bb2cf6c2335f40a51 (DEFAULT_TABLE table=test-table [id=2157b825341f4036b7ba766f01ef3305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:56:31.748298 3555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74b4aba12c0482bb2cf6c2335f40a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:56:31.748410 3421 tablet_service.cc:1467] Processing CreateTablet for tablet b74b4aba12c0482bb2cf6c2335f40a51 (DEFAULT_TABLE table=test-table [id=2157b825341f4036b7ba766f01ef3305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:56:31.748400 3689 tablet_service.cc:1467] Processing CreateTablet for tablet b74b4aba12c0482bb2cf6c2335f40a51 (DEFAULT_TABLE table=test-table [id=2157b825341f4036b7ba766f01ef3305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:56:31.749748 3689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74b4aba12c0482bb2cf6c2335f40a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:56:31.749758 3421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74b4aba12c0482bb2cf6c2335f40a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:56:31.759958 3773 tablet_bootstrap.cc:492] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8: Bootstrap starting.
I20260926 02:56:31.763449 3773 tablet_bootstrap.cc:654] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:31.764709 3774 tablet_bootstrap.cc:492] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76: Bootstrap starting.
I20260926 02:56:31.764901 3773 log.cc:824] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:31.766927 3775 tablet_bootstrap.cc:492] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818: Bootstrap starting.
I20260926 02:56:31.769111 3774 tablet_bootstrap.cc:654] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:31.769975 3775 tablet_bootstrap.cc:654] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:31.770648 3774 log.cc:824] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:31.771072 3775 log.cc:824] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:31.775218 3773 tablet_bootstrap.cc:492] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8: No bootstrap required, opened a new log
I20260926 02:56:31.775461 3775 tablet_bootstrap.cc:492] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818: No bootstrap required, opened a new log
I20260926 02:56:31.775564 3773 ts_tablet_manager.cc:1397] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260926 02:56:31.775759 3775 ts_tablet_manager.cc:1397] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260926 02:56:31.775827 3774 tablet_bootstrap.cc:492] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76: No bootstrap required, opened a new log
I20260926 02:56:31.776104 3774 ts_tablet_manager.cc:1397] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260926 02:56:31.782776 3775 raft_consensus.cc:348] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } } peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } }
I20260926 02:56:31.783437 3775 raft_consensus.cc:374] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:31.783603 3775 raft_consensus.cc:729] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1ff144506a47f0876a360d9728c818, State: Initialized, Role: FOLLOWER
I20260926 02:56:31.784482 3775 consensus_queue.cc:260] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } } peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } }
I20260926 02:56:31.786556 3775 ts_tablet_manager.cc:1428] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20260926 02:56:31.788515 3774 raft_consensus.cc:348] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } } peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } }
I20260926 02:56:31.789048 3774 raft_consensus.cc:374] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:31.789202 3774 raft_consensus.cc:729] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c26acfcdf844eb7b8476d1aa2018f76, State: Initialized, Role: FOLLOWER
I20260926 02:56:31.789402 3773 raft_consensus.cc:348] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } } peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } }
I20260926 02:56:31.789872 3773 raft_consensus.cc:374] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:31.789981 3773 raft_consensus.cc:729] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d5768070d84037a35f0ba378c5e3f8, State: Initialized, Role: FOLLOWER
I20260926 02:56:31.790041 3774 consensus_queue.cc:260] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } } peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } }
I20260926 02:56:31.790850 3773 consensus_queue.cc:260] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } } peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } }
I20260926 02:56:31.791260 3754 heartbeater.cc:503] Master 127.19.185.254:38225 was elected leader, sending a full tablet report...
I20260926 02:56:31.792201 3774 ts_tablet_manager.cc:1428] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260926 02:56:31.793257 3620 heartbeater.cc:503] Master 127.19.185.254:38225 was elected leader, sending a full tablet report...
I20260926 02:56:31.794028 3773 ts_tablet_manager.cc:1428] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8: Time spent starting tablet: real 0.018s user 0.014s sys 0.002s
I20260926 02:56:31.809155 3780 raft_consensus.cc:482] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:56:31.809442 3780 raft_consensus.cc:504] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } } peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } }
I20260926 02:56:31.811167 3780 leader_election.cc:288] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94d5768070d84037a35f0ba378c5e3f8 (127.19.185.194:34651), 6b1ff144506a47f0876a360d9728c818 (127.19.185.193:46763)
I20260926 02:56:31.817220 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.811475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:34651 (local address 127.19.185.195:44429)
0926 02:56:31.811888 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:31.811918 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:31.812065 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:31.812655 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:31.812660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:31.812693 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:31.812946 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:31.812960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.813973 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.813980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:31.815359 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:31.815374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.816427 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.816433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:31.816620 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.816644 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:56:31.816777 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 02:56:31.816902 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":1,"spinlock_wait_cycles":34560}
I20260926 02:56:31.817262 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.811487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:44429 (local address 127.19.185.194:34651)
0926 02:56:31.811692 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:31.811698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:31.812001 (+ 303us) server_negotiation.cc:407] Connection header received
0926 02:56:31.812262 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:31.812271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:31.812389 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:31.812500 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:31.813095 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.813841 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.815510 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.816276 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.816436 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.816507 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:31.816851 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 02:56:31.816999 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:56:31.818068 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.811673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:53293 (local address 127.19.185.193:46763)
0926 02:56:31.811857 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:31.811862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:31.812780 (+ 918us) server_negotiation.cc:407] Connection header received
0926 02:56:31.812987 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:31.812993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:31.813069 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:31.813156 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:31.813684 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.814375 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.816108 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.816973 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.817122 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.817189 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:31.817669 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 02:56:31.817872 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 02:56:31.818717 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.811686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46763 (local address 127.19.185.195:53293)
0926 02:56:31.812657 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:31.812679 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:31.812827 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:31.813191 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:31.813196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:31.813219 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:31.813536 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:31.813544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.814533 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.814539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:31.815952 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:31.815964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.817173 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.817185 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:31.817487 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.817518 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:56:31.818272 (+ 754us) client_negotiation.cc:240] Negotiation successful
0926 02:56:31.818408 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"spinlock_wait_cycles":7296,"thread_start_us":682,"threads_started":1}
I20260926 02:56:31.818634 3575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74b4aba12c0482bb2cf6c2335f40a51" candidate_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d5768070d84037a35f0ba378c5e3f8" is_pre_election: true
I20260926 02:56:31.819137 3575 raft_consensus.cc:2454] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c26acfcdf844eb7b8476d1aa2018f76 in term 0.
I20260926 02:56:31.819614 3441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74b4aba12c0482bb2cf6c2335f40a51" candidate_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1ff144506a47f0876a360d9728c818" is_pre_election: true
I20260926 02:56:31.819831 3642 leader_election.cc:302] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94d5768070d84037a35f0ba378c5e3f8, 9c26acfcdf844eb7b8476d1aa2018f76; no voters:
I20260926 02:56:31.820125 3441 raft_consensus.cc:2454] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c26acfcdf844eb7b8476d1aa2018f76 in term 0.
I20260926 02:56:31.820237 3780 raft_consensus.cc:2781] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:56:31.820385 3780 raft_consensus.cc:482] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:56:31.820508 3780 raft_consensus.cc:3037] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:56:31.822708 3780 raft_consensus.cc:504] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } } peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } }
I20260926 02:56:31.823487 3780 leader_election.cc:288] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [CANDIDATE]: Term 1 election: Requested vote from peers 94d5768070d84037a35f0ba378c5e3f8 (127.19.185.194:34651), 6b1ff144506a47f0876a360d9728c818 (127.19.185.193:46763)
I20260926 02:56:31.823840 3575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74b4aba12c0482bb2cf6c2335f40a51" candidate_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d5768070d84037a35f0ba378c5e3f8"
I20260926 02:56:31.824054 3575 raft_consensus.cc:3037] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:56:31.824054 3441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74b4aba12c0482bb2cf6c2335f40a51" candidate_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1ff144506a47f0876a360d9728c818"
I20260926 02:56:31.824293 3441 raft_consensus.cc:3037] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:56:31.826270 3575 raft_consensus.cc:2454] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c26acfcdf844eb7b8476d1aa2018f76 in term 1.
I20260926 02:56:31.826653 3642 leader_election.cc:302] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94d5768070d84037a35f0ba378c5e3f8, 9c26acfcdf844eb7b8476d1aa2018f76; no voters:
I20260926 02:56:31.826931 3780 raft_consensus.cc:2781] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:56:31.827087 3441 raft_consensus.cc:2454] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c26acfcdf844eb7b8476d1aa2018f76 in term 1.
I20260926 02:56:31.827538 3780 raft_consensus.cc:686] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 1 LEADER]: Becoming Leader. State: Replica: 9c26acfcdf844eb7b8476d1aa2018f76, State: Running, Role: LEADER
I20260926 02:56:31.828162 3780 consensus_queue.cc:237] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } } peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } }
I20260926 02:56:31.833249 3297 catalog_manager.cc:5495] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c26acfcdf844eb7b8476d1aa2018f76 (127.19.185.195). New cstate: current_term: 1 leader_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } health_report { overall_health: HEALTHY } } }
W20260926 02:56:31.844837 3487 tablet.cc:2369] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:56:31.869776 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.864352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46914 (local address 127.19.185.195:35819)
0926 02:56:31.864584 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:31.864589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:31.864609 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:56:31.864853 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:31.864857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:31.864941 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:31.865074 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:56:31.865448 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.866166 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.867574 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.867964 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.868057 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.868587 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:56:31.869091 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:56:31.869202 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:31.869371 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 02:56:31.869452 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:56:31.878329 3441 raft_consensus.cc:1261] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818 [term 1 FOLLOWER]: Refusing update from remote peer 9c26acfcdf844eb7b8476d1aa2018f76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:56:31.878326 3575 raft_consensus.cc:1261] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8 [term 1 FOLLOWER]: Refusing update from remote peer 9c26acfcdf844eb7b8476d1aa2018f76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:56:31.879364 3783 consensus_queue.cc:1035] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:56:31.879711 3780 consensus_queue.cc:1035] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 02:56:31.886466 3621 tablet.cc:2369] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:56:31.894834 3788 mvcc.cc:205] Tried to move back new op lower bound from 7333443141123670016 to 7333443140932407296. Current Snapshot: MvccSnapshot[applied={T|T < 7333443141123670016}]
I20260926 02:56:31.908875 3296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:56:31.909222 3296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:56:31.913271 3296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Sending DeleteTablet for 3 replicas of tablet b74b4aba12c0482bb2cf6c2335f40a51
I20260926 02:56:31.914494 3421 tablet_service.cc:1514] Processing DeleteTablet for tablet b74b4aba12c0482bb2cf6c2335f40a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:56:31 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:53478
I20260926 02:56:31.914692 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 3358
I20260926 02:56:31.914757 3555 tablet_service.cc:1514] Processing DeleteTablet for tablet b74b4aba12c0482bb2cf6c2335f40a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:56:31 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:45952
I20260926 02:56:31.915530 3689 tablet_service.cc:1514] Processing DeleteTablet for tablet b74b4aba12c0482bb2cf6c2335f40a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:56:31 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:46900
I20260926 02:56:31.928613 3796 tablet_replica.cc:330] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818: stopping tablet replica
I20260926 02:56:31.929704 3799 tablet_replica.cc:330] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76: stopping tablet replica
I20260926 02:56:31.930012 3796 raft_consensus.cc:2229] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:56:31.930132 3799 raft_consensus.cc:2229] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:56:31.930692 3796 raft_consensus.cc:2258] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:56:31.930822 3799 raft_consensus.cc:2258] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:56:31.931628 3797 tablet_replica.cc:330] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8: stopping tablet replica
I20260926 02:56:31.932097 3797 raft_consensus.cc:2229] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:56:31.932631 3797 raft_consensus.cc:2258] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:56:31.933312 3799 ts_tablet_manager.cc:1905] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:56:31.933524 3796 ts_tablet_manager.cc:1905] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:56:31.937182 3797 ts_tablet_manager.cc:1905] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:56:31.940898 3799 ts_tablet_manager.cc:1918] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:56:31.941232 3799 log.cc:1196] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/b74b4aba12c0482bb2cf6c2335f40a51
I20260926 02:56:31.941907 3799 ts_tablet_manager.cc:1939] T b74b4aba12c0482bb2cf6c2335f40a51 P 9c26acfcdf844eb7b8476d1aa2018f76: Deleting consensus metadata
I20260926 02:56:31.942193 3796 ts_tablet_manager.cc:1918] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 26 02:56:31 dist-test-slave-ktng krb5kdc[3252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:56:31.942435 3796 log.cc:1196] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/b74b4aba12c0482bb2cf6c2335f40a51
I20260926 02:56:31.943085 3796 ts_tablet_manager.cc:1939] T b74b4aba12c0482bb2cf6c2335f40a51 P 6b1ff144506a47f0876a360d9728c818: Deleting consensus metadata
I20260926 02:56:31.944098 3281 catalog_manager.cc:4841] TS 9c26acfcdf844eb7b8476d1aa2018f76 (127.19.185.195:35819): tablet b74b4aba12c0482bb2cf6c2335f40a51 (table test-table [id=2157b825341f4036b7ba766f01ef3305]) successfully deleted
I20260926 02:56:31.945482 3283 catalog_manager.cc:4841] TS 6b1ff144506a47f0876a360d9728c818 (127.19.185.193:46763): tablet b74b4aba12c0482bb2cf6c2335f40a51 (table test-table [id=2157b825341f4036b7ba766f01ef3305]) successfully deleted
I20260926 02:56:31.947736 3797 ts_tablet_manager.cc:1918] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:56:31.947921 3797 log.cc:1196] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/b74b4aba12c0482bb2cf6c2335f40a51
I20260926 02:56:31.948503 3797 ts_tablet_manager.cc:1939] T b74b4aba12c0482bb2cf6c2335f40a51 P 94d5768070d84037a35f0ba378c5e3f8: Deleting consensus metadata
I20260926 02:56:31.950213 3281 catalog_manager.cc:4841] TS 94d5768070d84037a35f0ba378c5e3f8 (127.19.185.194:34651): tablet b74b4aba12c0482bb2cf6c2335f40a51 (table test-table [id=2157b825341f4036b7ba766f01ef3305]) successfully deleted
I20260926 02:56:31.951431 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:31.916036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.19.185.193:46763)
0926 02:56:31.916217 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:31.916224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:31.928934 (+ 12710us) server_negotiation.cc:407] Connection header received
0926 02:56:31.929195 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:31.929201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:31.929291 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:31.929425 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:31.933574 (+ 4149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.934663 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.936773 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:31.937452 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:31.937575 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:31.943673 (+ 6098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:31.943687 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:31.943708 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:31.943751 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:31.947839 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:31.948865 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:31.948875 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:31.948879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:31.949048 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:31.949509 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:31.949514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:31.949517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:31.949998 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:31.950295 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:31.950757 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 02:56:31.950984 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:56:31.952190 3481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:32.042717 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 3358
I20260926 02:56:32.067255 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 3492
Sep 26 02:56:32 dist-test-slave-ktng krb5kdc[3252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:56:32.086019 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:32.068260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.19.185.194:34651)
0926 02:56:32.068453 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:32.068457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:32.068478 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:56:32.068637 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:32.068641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:32.068709 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:32.068812 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:32.070479 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:32.071382 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:32.073206 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:32.073868 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:32.073998 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:32.078747 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:32.078765 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:32.078786 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:32.078836 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:32.082811 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:32.083629 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:32.083638 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:32.083642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:32.083794 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:32.084187 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:32.084192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:32.084196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:32.084704 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:32.084999 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:32.085427 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 02:56:32.085648 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:56:32.086831 3615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:32.177917 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 3492
I20260926 02:56:32.201637 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 3626
Sep 26 02:56:32 dist-test-slave-ktng krb5kdc[3252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:56:32.223040 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:32.202658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.19.185.195:35819)
0926 02:56:32.202829 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:32.202833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:32.202851 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:56:32.203029 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:32.203033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:32.203120 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:32.203205 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:32.205302 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:32.206173 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:32.208335 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:32.208824 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:32.208934 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:32.214901 (+ 5967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:32.214917 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:32.214940 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:32.215022 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:32.219658 (+ 4636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:32.220414 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:32.220425 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:32.220429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:32.220655 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:32.221132 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:32.221138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:32.221142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:32.221671 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:32.222043 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:32.222283 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 02:56:32.222538 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:56:32.223899 3749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:32.347888 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 3626
I20260926 02:56:32.364189 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 3265
I20260926 02:56:32.365657 3327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:32.478607 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 3265
I20260926 02:56:52.501653 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:38225
--webserver_interface=127.19.185.254
--webserver_port=38933
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:38225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:56:52.837522 3804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:52.837893 3804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:52.838042 3804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:52.845211 3804 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 02:56:52.845345 3804 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 02:56:52.845438 3804 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 02:56:52.845530 3804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:56:52.845628 3804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:52.845749 3804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:52.845834 3804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:56:52.845914 3804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:56:52.855842 3804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:38225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:38225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=38933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:52.857172 3804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:52.858561 3804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:52.881681 3811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:52.881708 3813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:52.881772 3810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:52.882355 3804 server_base.cc:1034] running on GCE node
I20260926 02:56:52.883244 3804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:52.884807 3804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:52.886029 3804 hybrid_clock.cc:648] HybridClock initialized: now 1790391412885965 us; error 33 us; skew 500 ppm
I20260926 02:56:52.886374 3804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:52 dist-test-slave-ktng krb5kdc[3252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391412, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:52.889923 3804 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:56:52.891538 3804 webserver.cc:460] Webserver started at http://127.19.185.254:38933/ using document root <none> and password file <none>
I20260926 02:56:52.892292 3804 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:52.892470 3804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:52.898617 3804 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 02:56:52.901707 3820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:52.903220 3804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:56:52.903446 3804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "c15a442b8f874e159320957ea107c032"
format_stamp: "Formatted at 2026-09-26 02:56:29 on dist-test-slave-ktng"
server_key: "3086d6661544b8fdea6ddb41926597c4"
server_key_iv: "39258ae319e978679bec1f10d38e4726"
server_key_version: "encryptionkey@0"
I20260926 02:56:52.904493 3804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:52.937592 3804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:52.942297 3804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:52.942633 3804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:52.962638 3804 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:38225
I20260926 02:56:52.962663 3871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:38225 every 8 connection(s)
I20260926 02:56:52.964298 3804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:56:52.966588 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 3804
I20260926 02:56:52.967545 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:46763
--local_ip_for_outbound_sockets=127.19.185.193
--tserver_master_addrs=127.19.185.254:38225
--webserver_port=43913
--webserver_interface=127.19.185.193
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:56:52.971753 3872 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:56:52.975045 3872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Bootstrap starting.
I20260926 02:56:52.995414 3872 log.cc:824] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:53.008009 3872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:56:53.008689 3872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Bootstrap complete.
I20260926 02:56:53.018124 3872 raft_consensus.cc:348] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } }
I20260926 02:56:53.018985 3872 raft_consensus.cc:729] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15a442b8f874e159320957ea107c032, State: Initialized, Role: FOLLOWER
I20260926 02:56:53.019730 3872 consensus_queue.cc:260] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } }
I20260926 02:56:53.019980 3872 raft_consensus.cc:388] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:56:53.020114 3872 raft_consensus.cc:482] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:56:53.020301 3872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:56:53.024396 3872 raft_consensus.cc:504] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } }
I20260926 02:56:53.024922 3872 leader_election.cc:302] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c15a442b8f874e159320957ea107c032; no voters:
I20260926 02:56:53.025631 3872 leader_election.cc:288] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:56:53.025772 3876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:56:53.026973 3876 raft_consensus.cc:686] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 2 LEADER]: Becoming Leader. State: Replica: c15a442b8f874e159320957ea107c032, State: Running, Role: LEADER
I20260926 02:56:53.027637 3876 consensus_queue.cc:237] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } }
I20260926 02:56:53.028083 3872 sys_catalog.cc:564] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:56:53.030434 3878 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c15a442b8f874e159320957ea107c032. Latest consensus state: current_term: 2 leader_uuid: "c15a442b8f874e159320957ea107c032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } } }
I20260926 02:56:53.030820 3878 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: This master's current role is: LEADER
I20260926 02:56:53.030578 3877 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c15a442b8f874e159320957ea107c032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } } }
I20260926 02:56:53.031028 3877 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: This master's current role is: LEADER
I20260926 02:56:53.040730 3883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:56:53.051355 3883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:56:53.053346 3883 catalog_manager.cc:1259] Loaded cluster ID: b1e8173b89804c2fa71a9bed1486c255
I20260926 02:56:53.053574 3883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:56:53.059216 3883 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:56:53.061419 3883 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Loaded TSK: 0
I20260926 02:56:53.113427 3883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Generated new TSK 1
I20260926 02:56:53.113852 3883 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:56:53.339032 3874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:53.339402 3874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:53.339596 3874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:53.347096 3874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:53.347216 3874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:53.347433 3874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:56:53.357581 3874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:46763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=43913
--tserver_master_addrs=127.19.185.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:53.358959 3874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:53.360354 3874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:53.375792 3902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:53.375794 3900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:53.376387 3899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:53.376875 3874 server_base.cc:1034] running on GCE node
I20260926 02:56:53.377638 3874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:53.378816 3874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:53.380064 3874 hybrid_clock.cc:648] HybridClock initialized: now 1790391413379986 us; error 58 us; skew 500 ppm
I20260926 02:56:53.380458 3874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:53 dist-test-slave-ktng krb5kdc[3252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391413, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:53.384595 3874 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:56:53.386376 3874 webserver.cc:460] Webserver started at http://127.19.185.193:43913/ using document root <none> and password file <none>
I20260926 02:56:53.387375 3874 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:53.387537 3874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:53.395040 3874 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:56:53.398439 3909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:53.400157 3874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:56:53.400350 3874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "6b1ff144506a47f0876a360d9728c818"
format_stamp: "Formatted at 2026-09-26 02:56:30 on dist-test-slave-ktng"
server_key: "2dbe18d8c82c8ac975c8933c5128afce"
server_key_iv: "5e0791b026124e75cec065f8920ffb2a"
server_key_version: "encryptionkey@0"
I20260926 02:56:53.401736 3874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:53.425935 3874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:53.431756 3874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:53.432046 3874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:53.433164 3874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:53.434659 3874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:56:53.434747 3874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:53.434893 3874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:53.434984 3874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:53.474589 3874 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:46763
I20260926 02:56:53.474639 4021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:46763 every 8 connection(s)
I20260926 02:56:53.476032 3874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:56:53.478907 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 3874
I20260926 02:56:53.479707 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:34651
--local_ip_for_outbound_sockets=127.19.185.194
--tserver_master_addrs=127.19.185.254:38225
--webserver_port=41671
--webserver_interface=127.19.185.194
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:56:53 dist-test-slave-ktng krb5kdc[3252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391413, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:53.504686 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:53.479432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:50045)
0926 02:56:53.480345 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:53.480400 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:53.483116 (+ 2716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:53.484730 (+ 1614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:53.484742 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:53.485550 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:56:53.486331 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:53.486358 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:53.488380 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:53.488385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:53.489827 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:53.489841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:53.491474 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:53.491481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:53.491735 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:53.492511 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:56:53.492540 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:56:53.495426 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:56:53.500396 (+ 4970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:53.500404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:53.500418 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:53.500877 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:53.501413 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:53.501416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:53.501418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:53.501590 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:53.502635 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:56:53.502641 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:56:53.502941 (+ 300us) client_negotiation.cc:769] Sending connection context
0926 02:56:53.503174 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 02:56:53.503676 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":359,"threads_started":1}
I20260926 02:56:53.505343 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:53.479076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:50045 (local address 127.19.185.254:38225)
0926 02:56:53.479754 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:53.479775 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:53.480543 (+ 768us) server_negotiation.cc:407] Connection header received
0926 02:56:53.483781 (+ 3238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:53.483795 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:53.484184 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:53.484512 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:53.486578 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:53.488165 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:53.489988 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:53.491157 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:53.491411 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:53.495741 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:53.495753 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:53.495774 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:53.495834 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:53.500208 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:53.501026 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:53.501039 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:53.501048 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:53.501293 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:53.501698 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:53.501706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:53.501712 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:53.502479 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:53.502696 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:53.503480 (+ 784us) server_negotiation.cc:299] Negotiation successful
0926 02:56:53.503977 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":301,"threads_started":1}
I20260926 02:56:53.508282 4022 heartbeater.cc:347] Connected to a master server at 127.19.185.254:38225
I20260926 02:56:53.508631 4022 heartbeater.cc:464] Registering TS with master...
I20260926 02:56:53.509455 4022 heartbeater.cc:511] Master 127.19.185.254:38225 requested a full tablet report, sending...
I20260926 02:56:53.511669 3837 ts_manager.cc:194] Registered new tserver with Master: 6b1ff144506a47f0876a360d9728c818 (127.19.185.193:46763)
I20260926 02:56:53.513180 3837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:50045
W20260926 02:56:53.816733 4026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:53.817019 4026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:53.817166 4026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:53.825536 4026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:53.825663 4026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:53.825865 4026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:56:53.836197 4026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:34651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=41671
--tserver_master_addrs=127.19.185.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:53.837607 4026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:53.838956 4026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:53.852257 4031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:53.852233 4032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:53.852523 4034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:53.853061 4026 server_base.cc:1034] running on GCE node
I20260926 02:56:53.853652 4026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:53.854621 4026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:53.855832 4026 hybrid_clock.cc:648] HybridClock initialized: now 1790391413855782 us; error 46 us; skew 500 ppm
I20260926 02:56:53.856175 4026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:53 dist-test-slave-ktng krb5kdc[3252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391413, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:53.859268 4026 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:56:53.860746 4026 webserver.cc:460] Webserver started at http://127.19.185.194:41671/ using document root <none> and password file <none>
I20260926 02:56:53.861438 4026 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:53.861591 4026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:53.866894 4026 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:56:53.869591 4041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:53.870895 4026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 02:56:53.871070 4026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "94d5768070d84037a35f0ba378c5e3f8"
format_stamp: "Formatted at 2026-09-26 02:56:31 on dist-test-slave-ktng"
server_key: "e68ca22310fe5fbf5278953256ca82ed"
server_key_iv: "c436552ef6ea009dd05a32ddea31f919"
server_key_version: "encryptionkey@0"
I20260926 02:56:53.872129 4026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:53.886143 4026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:53.890136 4026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:53.890410 4026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:53.891381 4026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:53.892849 4026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:56:53.892976 4026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:53.893101 4026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:53.893187 4026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:53.927198 4026 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:34651
I20260926 02:56:53.927253 4153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:34651 every 8 connection(s)
I20260926 02:56:53.928907 4026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:56:53.938884 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 4026
I20260926 02:56:53.939565 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:35819
--local_ip_for_outbound_sockets=127.19.185.195
--tserver_master_addrs=127.19.185.254:38225
--webserver_port=32891
--webserver_interface=127.19.185.195
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:56:53 dist-test-slave-ktng krb5kdc[3252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391413, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:53.949247 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:53.931475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:35945 (local address 127.19.185.254:38225)
0926 02:56:53.931653 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:53.931658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:53.932862 (+ 1204us) server_negotiation.cc:407] Connection header received
0926 02:56:53.934701 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:53.934706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:53.934802 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:53.934911 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:53.936486 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:53.937235 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:53.938820 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:53.939375 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:53.939503 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:53.942937 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:53.942948 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:53.942952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:53.942983 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:53.945780 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:53.946524 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:53.946530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:53.946533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:53.946681 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:53.947202 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:53.947210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:53.947217 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:53.947673 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:53.947841 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:53.948537 (+ 696us) server_negotiation.cc:299] Negotiation successful
0926 02:56:53.948805 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:56:53.950235 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:53.931928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:35945)
0926 02:56:53.932643 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:53.932682 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:53.934433 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:53.935188 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:53.935196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:53.935756 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:56:53.936316 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:53.936335 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:53.937393 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:53.937403 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:53.938670 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:53.938682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:53.939591 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:53.939599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:53.939899 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:53.940568 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:56:53.940591 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:56:53.942700 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:56:53.945937 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:53.945946 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:53.945964 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:53.946400 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:53.946828 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:53.946833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:53.946836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:53.947036 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:53.947854 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:56:53.947860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:56:53.948172 (+ 312us) client_negotiation.cc:769] Sending connection context
0926 02:56:53.948410 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 02:56:53.948890 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":286,"threads_started":1}
I20260926 02:56:53.951894 4154 heartbeater.cc:347] Connected to a master server at 127.19.185.254:38225
I20260926 02:56:53.952188 4154 heartbeater.cc:464] Registering TS with master...
I20260926 02:56:53.952940 4154 heartbeater.cc:511] Master 127.19.185.254:38225 requested a full tablet report, sending...
I20260926 02:56:53.954488 3837 ts_manager.cc:194] Registered new tserver with Master: 94d5768070d84037a35f0ba378c5e3f8 (127.19.185.194:34651)
I20260926 02:56:53.955775 3837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:35945
W20260926 02:56:54.270426 4157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:56:54.270712 4157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:56:54.270823 4157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:56:54.278232 4157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:56:54.278335 4157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:56:54.278522 4157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:56:54.288254 4157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:35819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=32891
--tserver_master_addrs=127.19.185.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:56:54.289471 4157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:56:54.290736 4157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:56:54.303794 4162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:54.303915 4165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:56:54.303825 4163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:56:54.305387 4157 server_base.cc:1034] running on GCE node
I20260926 02:56:54.305963 4157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:56:54.306943 4157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:56:54.308145 4157 hybrid_clock.cc:648] HybridClock initialized: now 1790391414308092 us; error 40 us; skew 500 ppm
I20260926 02:56:54.308540 4157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:56:54 dist-test-slave-ktng krb5kdc[3252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391414, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:56:54.311755 4157 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:56:54.313105 4157 webserver.cc:460] Webserver started at http://127.19.185.195:32891/ using document root <none> and password file <none>
I20260926 02:56:54.313809 4157 fs_manager.cc:359] Metadata directory not provided
I20260926 02:56:54.313963 4157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:56:54.319248 4157 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:56:54.322119 4172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:54.323632 4157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:56:54.323812 4157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "9c26acfcdf844eb7b8476d1aa2018f76"
format_stamp: "Formatted at 2026-09-26 02:56:31 on dist-test-slave-ktng"
server_key: "60d0ab52cafc7c2b8bf33cc1d391ecf5"
server_key_iv: "565747288b75dac7d8801f55ec9728e3"
server_key_version: "encryptionkey@0"
I20260926 02:56:54.324874 4157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:56:54.341327 4157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:56:54.345446 4157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:56:54.345726 4157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:56:54.346694 4157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:56:54.348078 4157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:56:54.348193 4157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:54.348318 4157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:56:54.348407 4157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:56:54.381165 4157 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:35819
I20260926 02:56:54.381246 4284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:35819 every 8 connection(s)
I20260926 02:56:54.382570 4157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:56:54.384825 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 4157
Sep 26 02:56:54 dist-test-slave-ktng krb5kdc[3252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391414, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:56:54.400947 4287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.385715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.19.185.254:38225)
0926 02:56:54.386203 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:54.386211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:54.386653 (+ 442us) server_negotiation.cc:407] Connection header received
0926 02:56:54.386890 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:54.386896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:54.386985 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:54.387105 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:54.389232 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.389965 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.391284 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.391810 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.391950 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.394749 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:54.394754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:54.394757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:54.394789 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:54.397855 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:54.398594 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:54.398598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:54.398600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:54.398770 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:54.399278 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:54.399285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:54.399288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:54.399634 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:54.399838 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:54.400330 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 02:56:54.400530 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":241,"threads_started":1}
I20260926 02:56:54.405136 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.385145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:34277 (local address 127.19.185.254:38225)
0926 02:56:54.385354 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:54.385361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:54.387134 (+ 1773us) server_negotiation.cc:407] Connection header received
0926 02:56:54.389772 (+ 2638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:54.389778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:54.389855 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:54.389966 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:54.391459 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.392222 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.394006 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.394427 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.394553 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.398504 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:54.398510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:54.398514 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:54.398553 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:54.402001 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:54.402936 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:54.402941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:54.402944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:54.403097 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:54.403575 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:54.403580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:54.403583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:54.403922 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:54.404071 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:54.404714 (+ 643us) server_negotiation.cc:299] Negotiation successful
0926 02:56:54.404885 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:56:54.406095 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.385693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:34277)
0926 02:56:54.386979 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:54.387022 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:54.389519 (+ 2497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:54.390198 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:54.390206 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:54.390700 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:56:54.391249 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:54.391266 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.392387 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.392395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:54.393865 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:54.393878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.394668 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.394677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:54.394913 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.395607 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:56:54.395638 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:56:54.398281 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:56:54.402153 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:54.402160 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:54.402174 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:54.402801 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:54.403210 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:56:54.403214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:56:54.403216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:56:54.403457 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:56:54.404056 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:56:54.404065 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:56:54.404392 (+ 327us) client_negotiation.cc:769] Sending connection context
0926 02:56:54.404613 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 02:56:54.404992 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"thread_start_us":778,"threads_started":1}
I20260926 02:56:54.407570 4285 heartbeater.cc:347] Connected to a master server at 127.19.185.254:38225
I20260926 02:56:54.407899 4285 heartbeater.cc:464] Registering TS with master...
I20260926 02:56:54.408679 4285 heartbeater.cc:511] Master 127.19.185.254:38225 requested a full tablet report, sending...
I20260926 02:56:54.410071 3837 ts_manager.cc:194] Registered new tserver with Master: 9c26acfcdf844eb7b8476d1aa2018f76 (127.19.185.195:35819)
I20260926 02:56:54.410848 3837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:34277
I20260926 02:56:54.416510 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:56:54.423014 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.417384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.19.185.254:38225)
0926 02:56:54.417542 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:54.417547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:54.418102 (+ 555us) server_negotiation.cc:407] Connection header received
0926 02:56:54.418276 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:54.418280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:54.418439 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:54.418562 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:56:54.419060 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.419914 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.421350 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.421739 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.421831 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.422294 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:56:54.422490 (+ 196us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 02:56:54.422790 (+ 300us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38482: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 02:56:54.423141 4024 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38482: token expired
W20260926 02:56:54.423678 3759 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.19.185.254:38225)
I20260926 02:56:54.442800 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.424236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.19.185.254:38225)
0926 02:56:54.424398 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:54.424406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:54.424425 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:56:54.424505 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:54.424508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:54.424596 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:54.424696 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:54.426116 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.426815 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.428305 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.428721 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.428849 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.437247 (+ 8398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:54.437253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:54.437256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:54.437298 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:54.440021 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:54.440737 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:54.440743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:54.440752 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:54.440872 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:54.441253 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:54.441258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:54.441262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:54.441616 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:54.441872 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:54.442203 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 02:56:54.442454 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:56:54.446522 3837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:56:54.453970 3837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:56:54.494571 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.466751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.19.185.193:46763)
0926 02:56:54.482927 (+ 16176us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:54.482979 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:54.485157 (+ 2178us) server_negotiation.cc:407] Connection header received
0926 02:56:54.485266 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:54.485277 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:54.485542 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:54.485730 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:54.486883 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.489000 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.491419 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.492975 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.493170 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.493339 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:54.494000 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 02:56:54.494188 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15929,"thread_start_us":16223,"threads_started":1}
I20260926 02:56:54.494648 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.466051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46763 (local address 127.0.0.1:47270)
0926 02:56:54.484850 (+ 18799us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:54.484868 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:54.485004 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:54.486057 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:54.486068 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:54.486118 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:54.486634 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:54.486651 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.489196 (+ 2545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.489201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:54.491228 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:54.491240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.493234 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.493241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:54.493678 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.493742 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 02:56:54.493967 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 02:56:54.494186 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18667,"mutex_wait_us":86,"thread_start_us":18509,"threads_started":1}
I20260926 02:56:54.496829 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.485595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.19.185.194:34651)
0926 02:56:54.489382 (+ 3787us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:54.489394 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:54.489425 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:56:54.489532 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:54.489542 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:54.489767 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:54.490652 (+ 885us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:54.490859 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.492169 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.494492 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.495772 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.495978 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.496153 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:54.496360 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 02:56:54.496525 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3608,"thread_start_us":3504,"threads_started":1}
I20260926 02:56:54.496873 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.485394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:34651 (local address 127.0.0.1:32934)
0926 02:56:54.488604 (+ 3210us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:54.488623 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:54.488733 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:54.489985 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:54.489993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:54.490019 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:54.490453 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:54.490467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.492304 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.492309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:54.494299 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:54.494316 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.495988 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.495994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:54.496160 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.496200 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 02:56:54.496314 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 02:56:54.496456 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3082,"mutex_wait_us":61,"thread_start_us":3419,"threads_started":1}
I20260926 02:56:54.497058 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.465365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:35819 (local address 127.0.0.1:46552)
0926 02:56:54.484758 (+ 19393us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:54.484808 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:54.484990 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:54.488931 (+ 3941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:54.488935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:54.488976 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:54.489470 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:54.489484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.491798 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.491805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:54.494060 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:54.494074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.496216 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.496225 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:54.496402 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.496449 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 02:56:54.496515 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 02:56:54.496645 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19106,"mutex_wait_us":140,"spinlock_wait_cycles":3968,"thread_start_us":19095,"threads_started":1}
I20260926 02:56:54.498169 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.465800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46552 (local address 127.19.185.195:35819)
0926 02:56:54.488314 (+ 22514us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:54.488325 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:54.488366 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:56:54.488476 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:54.488483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:54.488729 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:54.489666 (+ 937us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:54.489982 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.491636 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.494237 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.495966 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.497349 (+ 1383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.497543 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:54.497607 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 02:56:54.497778 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22324,"thread_start_us":22153,"threads_started":1}
I20260926 02:56:54.499127 3957 tablet_service.cc:1467] Processing CreateTablet for tablet 371f0bd2ba294ff2ad39e9a6aa6e7e58 (DEFAULT_TABLE table=test-table [id=88ab30ec8b34429eb00ef5357e6c1374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:56:54.500739 3957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371f0bd2ba294ff2ad39e9a6aa6e7e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:56:54.500662 4089 tablet_service.cc:1467] Processing CreateTablet for tablet 371f0bd2ba294ff2ad39e9a6aa6e7e58 (DEFAULT_TABLE table=test-table [id=88ab30ec8b34429eb00ef5357e6c1374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:56:54.501771 4089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371f0bd2ba294ff2ad39e9a6aa6e7e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:56:54.501715 4220 tablet_service.cc:1467] Processing CreateTablet for tablet 371f0bd2ba294ff2ad39e9a6aa6e7e58 (DEFAULT_TABLE table=test-table [id=88ab30ec8b34429eb00ef5357e6c1374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:56:54.503008 4220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371f0bd2ba294ff2ad39e9a6aa6e7e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:56:54.514533 4301 tablet_bootstrap.cc:492] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8: Bootstrap starting.
I20260926 02:56:54.516364 4022 heartbeater.cc:503] Master 127.19.185.254:38225 was elected leader, sending a full tablet report...
I20260926 02:56:54.517881 4303 tablet_bootstrap.cc:492] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76: Bootstrap starting.
I20260926 02:56:54.518352 4301 tablet_bootstrap.cc:654] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:54.519830 4301 log.cc:824] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:54.522045 4303 tablet_bootstrap.cc:654] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:54.523342 4301 tablet_bootstrap.cc:492] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8: No bootstrap required, opened a new log
I20260926 02:56:54.523347 4303 log.cc:824] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:54.523645 4301 ts_tablet_manager.cc:1397] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 02:56:54.523952 4302 tablet_bootstrap.cc:492] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818: Bootstrap starting.
I20260926 02:56:54.526954 4302 tablet_bootstrap.cc:654] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818: Neither blocks nor log segments found. Creating new log.
I20260926 02:56:54.528604 4302 log.cc:824] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818: Log is configured to *not* fsync() on all Append() calls
I20260926 02:56:54.530073 4303 tablet_bootstrap.cc:492] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76: No bootstrap required, opened a new log
I20260926 02:56:54.530386 4303 ts_tablet_manager.cc:1397] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260926 02:56:54.531602 4302 tablet_bootstrap.cc:492] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818: No bootstrap required, opened a new log
I20260926 02:56:54.531876 4302 ts_tablet_manager.cc:1397] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 02:56:54.535669 4301 raft_consensus.cc:348] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } } peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } }
I20260926 02:56:54.536134 4301 raft_consensus.cc:374] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:54.536271 4301 raft_consensus.cc:729] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d5768070d84037a35f0ba378c5e3f8, State: Initialized, Role: FOLLOWER
I20260926 02:56:54.536919 4301 consensus_queue.cc:260] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } } peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } }
I20260926 02:56:54.540094 4154 heartbeater.cc:503] Master 127.19.185.254:38225 was elected leader, sending a full tablet report...
I20260926 02:56:54.539860 4302 raft_consensus.cc:348] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } } peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } }
I20260926 02:56:54.540436 4302 raft_consensus.cc:374] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:54.540587 4302 raft_consensus.cc:729] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1ff144506a47f0876a360d9728c818, State: Initialized, Role: FOLLOWER
I20260926 02:56:54.541115 4301 ts_tablet_manager.cc:1428] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260926 02:56:54.541333 4302 consensus_queue.cc:260] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } } peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } }
I20260926 02:56:54.541465 4303 raft_consensus.cc:348] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } } peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } }
I20260926 02:56:54.541986 4303 raft_consensus.cc:374] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:56:54.542138 4303 raft_consensus.cc:729] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c26acfcdf844eb7b8476d1aa2018f76, State: Initialized, Role: FOLLOWER
I20260926 02:56:54.543103 4302 ts_tablet_manager.cc:1428] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260926 02:56:54.544643 4303 consensus_queue.cc:260] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } } peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } }
I20260926 02:56:54.545761 4285 heartbeater.cc:503] Master 127.19.185.254:38225 was elected leader, sending a full tablet report...
I20260926 02:56:54.547343 4303 ts_tablet_manager.cc:1428] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
W20260926 02:56:54.634356 4286 tablet.cc:2369] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:56:54.681037 4155 tablet.cc:2369] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:56:54.688720 4307 raft_consensus.cc:482] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:56:54.689047 4307 raft_consensus.cc:504] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } } peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } }
I20260926 02:56:54.690842 4307 leader_election.cc:288] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c26acfcdf844eb7b8476d1aa2018f76 (127.19.185.195:35819), 6b1ff144506a47f0876a360d9728c818 (127.19.185.193:46763)
I20260926 02:56:54.698724 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.691458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:52319 (local address 127.19.185.193:46763)
0926 02:56:54.691658 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:54.691665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:54.692512 (+ 847us) server_negotiation.cc:407] Connection header received
0926 02:56:54.692784 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:54.692791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:54.692883 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:54.692975 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:54.693589 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.694452 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.696538 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.697730 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.697871 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.697981 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:54.698319 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 02:56:54.698448 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:56:54.699039 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.691457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:59135 (local address 127.19.185.195:35819)
0926 02:56:54.691686 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:54.691696 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:54.692895 (+ 1199us) server_negotiation.cc:407] Connection header received
0926 02:56:54.693095 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:54.693102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:54.693182 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:54.693278 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:56:54.694021 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.694951 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.696993 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.697827 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.698011 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.698088 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:54.698643 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 02:56:54.698776 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:56:54.699221 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.691386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46763 (local address 127.19.185.194:52319)
0926 02:56:54.692336 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:54.692368 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:54.692582 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:54.693003 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:54.693009 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:54.693038 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:54.693445 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:54.693458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.694595 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.694600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:54.696348 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:54.696372 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.697970 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.697977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:54.698157 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.698180 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:56:54.698288 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 02:56:54.698782 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":1,"thread_start_us":481,"threads_started":1}
I20260926 02:56:54.699265 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.691990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:35819 (local address 127.19.185.194:59135)
0926 02:56:54.692783 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:54.692800 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:56:54.692959 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:56:54.693318 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:56:54.693328 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:56:54.693348 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:56:54.693811 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:54.693824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.695103 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.695107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:54.696787 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:56:54.696804 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.698074 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.698079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:56:54.698271 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.698334 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 02:56:54.698606 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 02:56:54.698737 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":3,"thread_start_us":437,"threads_started":1}
I20260926 02:56:54.700300 3977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371f0bd2ba294ff2ad39e9a6aa6e7e58" candidate_uuid: "94d5768070d84037a35f0ba378c5e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1ff144506a47f0876a360d9728c818" is_pre_election: true
I20260926 02:56:54.700378 4240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371f0bd2ba294ff2ad39e9a6aa6e7e58" candidate_uuid: "94d5768070d84037a35f0ba378c5e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" is_pre_election: true
I20260926 02:56:54.700838 3977 raft_consensus.cc:2454] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d5768070d84037a35f0ba378c5e3f8 in term 0.
I20260926 02:56:54.700870 4240 raft_consensus.cc:2454] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d5768070d84037a35f0ba378c5e3f8 in term 0.
I20260926 02:56:54.701545 4044 leader_election.cc:302] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b1ff144506a47f0876a360d9728c818, 94d5768070d84037a35f0ba378c5e3f8; no voters:
I20260926 02:56:54.701977 4307 raft_consensus.cc:2781] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:56:54.702128 4307 raft_consensus.cc:482] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:56:54.702240 4307 raft_consensus.cc:3037] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:56:54.704452 4307 raft_consensus.cc:504] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } } peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } }
I20260926 02:56:54.705058 4307 leader_election.cc:288] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [CANDIDATE]: Term 1 election: Requested vote from peers 9c26acfcdf844eb7b8476d1aa2018f76 (127.19.185.195:35819), 6b1ff144506a47f0876a360d9728c818 (127.19.185.193:46763)
I20260926 02:56:54.705595 4240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371f0bd2ba294ff2ad39e9a6aa6e7e58" candidate_uuid: "94d5768070d84037a35f0ba378c5e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c26acfcdf844eb7b8476d1aa2018f76"
I20260926 02:56:54.705595 3977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371f0bd2ba294ff2ad39e9a6aa6e7e58" candidate_uuid: "94d5768070d84037a35f0ba378c5e3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1ff144506a47f0876a360d9728c818"
I20260926 02:56:54.705811 4240 raft_consensus.cc:3037] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:56:54.705806 3977 raft_consensus.cc:3037] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:56:54.709185 3977 raft_consensus.cc:2454] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d5768070d84037a35f0ba378c5e3f8 in term 1.
I20260926 02:56:54.709185 4240 raft_consensus.cc:2454] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d5768070d84037a35f0ba378c5e3f8 in term 1.
I20260926 02:56:54.709620 4044 leader_election.cc:302] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b1ff144506a47f0876a360d9728c818, 94d5768070d84037a35f0ba378c5e3f8; no voters:
I20260926 02:56:54.709992 4307 raft_consensus.cc:2781] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:56:54.710510 4307 raft_consensus.cc:686] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 94d5768070d84037a35f0ba378c5e3f8, State: Running, Role: LEADER
I20260926 02:56:54.710985 4307 consensus_queue.cc:237] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } } peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } }
I20260926 02:56:54.716207 3837 catalog_manager.cc:5495] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94d5768070d84037a35f0ba378c5e3f8 (127.19.185.194). New cstate: current_term: 1 leader_uuid: "94d5768070d84037a35f0ba378c5e3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94d5768070d84037a35f0ba378c5e3f8" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 } health_report { overall_health: UNKNOWN } } }
W20260926 02:56:54.728533 4023 tablet.cc:2369] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:56:54.781538 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.775557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.19.185.194:34651)
0926 02:56:54.775774 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:54.775779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:54.775798 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:56:54.775976 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:54.775979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:54.776094 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:54.776217 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:56:54.776559 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.777452 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.778886 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.779646 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.779755 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.780369 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:56:54.780908 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:56:54.781026 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:54.781134 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 02:56:54.781202 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:56:54.789934 4240 raft_consensus.cc:1261] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 1 FOLLOWER]: Refusing update from remote peer 94d5768070d84037a35f0ba378c5e3f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:56:54.789932 3977 raft_consensus.cc:1261] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818 [term 1 FOLLOWER]: Refusing update from remote peer 94d5768070d84037a35f0ba378c5e3f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:56:54.790993 4312 consensus_queue.cc:1035] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c26acfcdf844eb7b8476d1aa2018f76" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:56:54.791374 4307 consensus_queue.cc:1035] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1ff144506a47f0876a360d9728c818" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:56:54.802134 4315 mvcc.cc:205] Tried to move back new op lower bound from 7333443234969436160 to 7333443234660540416. Current Snapshot: MvccSnapshot[applied={T|T < 7333443234969436160}]
I20260926 02:56:54.802446 4317 mvcc.cc:205] Tried to move back new op lower bound from 7333443234969436160 to 7333443234660540416. Current Snapshot: MvccSnapshot[applied={T|T < 7333443234969436160}]
I20260926 02:56:54.817487 3837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:56:54.817893 3837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:56:54.821636 3837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Sending DeleteTablet for 3 replicas of tablet 371f0bd2ba294ff2ad39e9a6aa6e7e58
I20260926 02:56:54.822788 4220 tablet_service.cc:1514] Processing DeleteTablet for tablet 371f0bd2ba294ff2ad39e9a6aa6e7e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:56:54 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:46552
I20260926 02:56:54.822979 4089 tablet_service.cc:1514] Processing DeleteTablet for tablet 371f0bd2ba294ff2ad39e9a6aa6e7e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:56:54 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:32934
I20260926 02:56:54.823016 3957 tablet_service.cc:1514] Processing DeleteTablet for tablet 371f0bd2ba294ff2ad39e9a6aa6e7e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:56:54 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:47270
I20260926 02:56:54.824105 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 3874
I20260926 02:56:54.835840 4326 tablet_replica.cc:330] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818: stopping tablet replica
I20260926 02:56:54.836449 4326 raft_consensus.cc:2229] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:56:54.837105 4326 raft_consensus.cc:2258] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:56:54.837409 4324 tablet_replica.cc:330] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76: stopping tablet replica
I20260926 02:56:54.838155 4324 raft_consensus.cc:2229] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:56:54.838655 4324 raft_consensus.cc:2258] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:56:54.839649 4326 ts_tablet_manager.cc:1905] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:56:54.839762 4325 tablet_replica.cc:330] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8: stopping tablet replica
I20260926 02:56:54.840306 4325 raft_consensus.cc:2229] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:56:54.840998 4325 raft_consensus.cc:2258] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:56:54.843199 4325 ts_tablet_manager.cc:1905] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:56:54.847150 4326 ts_tablet_manager.cc:1918] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:56:54.847354 4326 log.cc:1196] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/371f0bd2ba294ff2ad39e9a6aa6e7e58
I20260926 02:56:54.847925 4326 ts_tablet_manager.cc:1939] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 6b1ff144506a47f0876a360d9728c818: Deleting consensus metadata
I20260926 02:56:54.848066 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.825136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.19.185.193:46763)
0926 02:56:54.825301 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:54.825307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:54.825330 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:56:54.825547 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:54.825552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:54.825638 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:54.825745 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:54.827993 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.829237 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.831482 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.832286 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.832414 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.838894 (+ 6480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:54.838913 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:54.838932 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:54.838980 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:54.843104 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:54.844223 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:54.844233 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:54.844237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:54.844416 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:54.844962 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:54.844973 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:54.844976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:54.845502 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:54.845841 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:54.847368 (+ 1527us) server_negotiation.cc:299] Negotiation successful
0926 02:56:54.847650 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 02:56:54.848877 4017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:54.851256 4324 ts_tablet_manager.cc:1905] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:56:54.851771 4325 ts_tablet_manager.cc:1918] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:56:54.851981 4325 log.cc:1196] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/371f0bd2ba294ff2ad39e9a6aa6e7e58
I20260926 02:56:54.852658 4325 ts_tablet_manager.cc:1939] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 94d5768070d84037a35f0ba378c5e3f8: Deleting consensus metadata
I20260926 02:56:54.854894 3821 catalog_manager.cc:4841] TS 94d5768070d84037a35f0ba378c5e3f8 (127.19.185.194:34651): tablet 371f0bd2ba294ff2ad39e9a6aa6e7e58 (table test-table [id=88ab30ec8b34429eb00ef5357e6c1374]) successfully deleted
I20260926 02:56:54.859944 4324 ts_tablet_manager.cc:1918] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:56:54.860129 4324 log.cc:1196] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/371f0bd2ba294ff2ad39e9a6aa6e7e58
I20260926 02:56:54.860656 4324 ts_tablet_manager.cc:1939] T 371f0bd2ba294ff2ad39e9a6aa6e7e58 P 9c26acfcdf844eb7b8476d1aa2018f76: Deleting consensus metadata
I20260926 02:56:54.862625 3821 catalog_manager.cc:4841] TS 9c26acfcdf844eb7b8476d1aa2018f76 (127.19.185.195:35819): tablet 371f0bd2ba294ff2ad39e9a6aa6e7e58 (table test-table [id=88ab30ec8b34429eb00ef5357e6c1374]) successfully deleted
I20260926 02:56:54.946643 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 3874
W20260926 02:56:54.965849 3823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.185.193:46763 (error 108)
I20260926 02:56:54.966615 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 4026
I20260926 02:56:54.968518 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.968069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46763 (local address 127.0.0.1:47292)
0926 02:56:54.968226 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:54.968356 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 02:56:54.968784 3823 catalog_manager.cc:4584] TS 6b1ff144506a47f0876a360d9728c818 (127.19.185.193:46763): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 371f0bd2ba294ff2ad39e9a6aa6e7e58: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46763: connect: Connection refused (error 111)
I20260926 02:56:54.986848 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:54.967889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32952 (local address 127.19.185.194:34651)
0926 02:56:54.968049 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:54.968055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:54.968076 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:56:54.968144 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:54.968149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:54.968256 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:54.968362 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:54.970266 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.971478 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.973636 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:54.974422 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:54.974561 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:54.978417 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:54.978438 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:54.978458 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:54.978520 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:54.982928 (+ 4408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:54.983948 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:54.983959 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:54.983963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:54.984121 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:54.984595 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:54.984600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:54.984603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:54.985073 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:54.985461 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:54.985734 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 02:56:54.986434 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:56:54.987845 4149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:55.022521 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:55.021964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46763 (local address 127.0.0.1:47300)
0926 02:56:55.022204 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:55.022344 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 02:56:55.023710 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:55.023265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46763 (local address 127.0.0.1:47316)
0926 02:56:55.023474 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:55.023575 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 02:56:55.084380 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:55.083812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46763 (local address 127.0.0.1:47324)
0926 02:56:55.084047 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:55.084158 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 02:56:55.085552 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:55.085176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46763 (local address 127.0.0.1:47340)
0926 02:56:55.085340 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:55.085420 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 02:56:55.106719 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 4026
I20260926 02:56:55.125984 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 4157
I20260926 02:56:55.139722 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:55.126712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.19.185.195:35819)
0926 02:56:55.126893 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 02:56:55.126901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:56:55.126943 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:56:55.127134 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:56:55.127140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:56:55.127215 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:56:55.127335 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:56:55.128827 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:55.129638 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:55.130901 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:56:55.131333 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:56:55.131465 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:56:55.134058 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:56:55.134083 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:56:55.134100 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:56:55.134148 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:56:55.136836 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:55.137570 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:55.137580 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:55.137584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:55.137742 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:56:55.138097 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:56:55.138103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:56:55.138107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:56:55.138460 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:56:55.138747 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 02:56:55.139037 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 02:56:55.139251 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:56:55.140445 4280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:55.166515 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:55.166018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46763 (local address 127.0.0.1:47354)
0926 02:56:55.166202 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:55.166309 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 02:56:55.167713 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:55.167338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46763 (local address 127.0.0.1:47366)
0926 02:56:55.167513 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:55.167589 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 02:56:55.240986 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 4157
I20260926 02:56:55.257591 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 3804
I20260926 02:56:55.258755 3867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:56:55.374679 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:56:55.373717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46763 (local address 127.0.0.1:47382)
0926 02:56:55.374345 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0926 02:56:55.374442 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452}
I20260926 02:56:55.374634 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 3804
I20260926 02:57:15.397075 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:38225
--webserver_interface=127.19.185.254
--webserver_port=38933
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:38225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:15.735824 4332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:15.736191 4332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:15.736368 4332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:15.743716 4332 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 02:57:15.743872 4332 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 02:57:15.743945 4332 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 02:57:15.744022 4332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:57:15.744123 4332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:15.744290 4332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:15.744375 4332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:57:15.744458 4332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:57:15.754642 4332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:38225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:38225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=38933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:15.756120 4332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:15.757516 4332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:15.770004 4337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:15.769990 4338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:15.770447 4332 server_base.cc:1034] running on GCE node
W20260926 02:57:15.770026 4340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:15.771644 4332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:15.773087 4332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:15.774346 4332 hybrid_clock.cc:648] HybridClock initialized: now 1790391435774276 us; error 41 us; skew 500 ppm
I20260926 02:57:15.774667 4332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:15 dist-test-slave-ktng krb5kdc[3252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391435, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:15.783779 4332 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:57:15.785285 4332 webserver.cc:460] Webserver started at http://127.19.185.254:38933/ using document root <none> and password file <none>
I20260926 02:57:15.786002 4332 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:15.786088 4332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:15.791862 4332 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:57:15.794829 4347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:15.796299 4332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:57:15.796458 4332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "c15a442b8f874e159320957ea107c032"
format_stamp: "Formatted at 2026-09-26 02:56:29 on dist-test-slave-ktng"
server_key: "3086d6661544b8fdea6ddb41926597c4"
server_key_iv: "39258ae319e978679bec1f10d38e4726"
server_key_version: "encryptionkey@0"
I20260926 02:57:15.797384 4332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:15.812903 4332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:15.817411 4332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:15.817651 4332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:15.837138 4332 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:38225
I20260926 02:57:15.837180 4398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:38225 every 8 connection(s)
I20260926 02:57:15.838724 4332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:57:15.842012 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 4332
I20260926 02:57:15.843048 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:46763
--local_ip_for_outbound_sockets=127.19.185.193
--tserver_master_addrs=127.19.185.254:38225
--webserver_port=43913
--webserver_interface=127.19.185.193
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:57:15.847960 4399 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:57:15.853147 4399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Bootstrap starting.
I20260926 02:57:15.874238 4399 log.cc:824] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:15.890658 4399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:57:15.891225 4399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Bootstrap complete.
I20260926 02:57:15.901228 4399 raft_consensus.cc:348] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } }
I20260926 02:57:15.902288 4399 raft_consensus.cc:729] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15a442b8f874e159320957ea107c032, State: Initialized, Role: FOLLOWER
I20260926 02:57:15.902945 4399 consensus_queue.cc:260] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } }
I20260926 02:57:15.903218 4399 raft_consensus.cc:388] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:57:15.903367 4399 raft_consensus.cc:482] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:57:15.903571 4399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 2 FOLLOWER]: Advancing to term 3
I20260926 02:57:15.907382 4399 raft_consensus.cc:504] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } }
I20260926 02:57:15.907884 4399 leader_election.cc:302] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c15a442b8f874e159320957ea107c032; no voters:
I20260926 02:57:15.908592 4399 leader_election.cc:288] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 02:57:15.908725 4403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 02:57:15.909832 4403 raft_consensus.cc:686] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [term 3 LEADER]: Becoming Leader. State: Replica: c15a442b8f874e159320957ea107c032, State: Running, Role: LEADER
I20260926 02:57:15.910445 4403 consensus_queue.cc:237] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } }
I20260926 02:57:15.910825 4399 sys_catalog.cc:564] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:57:15.913069 4405 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c15a442b8f874e159320957ea107c032. Latest consensus state: current_term: 3 leader_uuid: "c15a442b8f874e159320957ea107c032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } } }
I20260926 02:57:15.913517 4405 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:15.914252 4404 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c15a442b8f874e159320957ea107c032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15a442b8f874e159320957ea107c032" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38225 } } }
I20260926 02:57:15.914602 4404 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032 [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:15.923815 4409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:57:15.934780 4409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:57:15.936872 4409 catalog_manager.cc:1259] Loaded cluster ID: b1e8173b89804c2fa71a9bed1486c255
I20260926 02:57:15.937103 4409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:57:15.941473 4420 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:57:15.942314 4409 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:57:15.943614 4409 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Loaded TSK: 0
I20260926 02:57:15.943753 4409 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c15a442b8f874e159320957ea107c032: Loaded TSK: 1
F20260926 02:57:15.944659 4409 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb9cffae9ad google::LogMessage::Fail() at ??:0
@ 0x7fb9cffb188a google::LogMessage::SendToLog() at ??:0
@ 0x7fb9cffae4de google::LogMessage::Flush() at ??:0
@ 0x7fb9cffafa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb9d3d127d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb9d3d0173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb9e59dd1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb9e59dc783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb9e5a682a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fb9e5a680c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb9e5ab25e9 std::function<>::operator()() at ??:0
@ 0x7fb9e59e0cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb9e59dec36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb9e5a66610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb9e5a66439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb9d2bfa345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb9d2bfd91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb9d2bfd799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb9d2bd8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb9d5eb26db start_thread at ??:0
@ 0x7fb9cee4861f clone at ??:0
W20260926 02:57:16.259342 4401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:16.259645 4401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:16.259816 4401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:16.267367 4401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:16.267475 4401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:16.267678 4401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:57:16.277482 4401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:46763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=43913
--tserver_master_addrs=127.19.185.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:16.278789 4401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:16.280165 4401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:16.293427 4449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:16.293511 4446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:16.293491 4447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:16.294420 4401 server_base.cc:1034] running on GCE node
I20260926 02:57:16.295027 4401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:16.296056 4401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:16.297273 4401 hybrid_clock.cc:648] HybridClock initialized: now 1790391436297230 us; error 36 us; skew 500 ppm
I20260926 02:57:16.297616 4401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:16 dist-test-slave-ktng krb5kdc[3252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391436, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:16.301069 4401 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:57:16.302652 4401 webserver.cc:460] Webserver started at http://127.19.185.193:43913/ using document root <none> and password file <none>
I20260926 02:57:16.303385 4401 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:16.303496 4401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:16.308938 4401 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:57:16.311869 4456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:16.313342 4401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:57:16.313529 4401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "6b1ff144506a47f0876a360d9728c818"
format_stamp: "Formatted at 2026-09-26 02:56:30 on dist-test-slave-ktng"
server_key: "2dbe18d8c82c8ac975c8933c5128afce"
server_key_iv: "5e0791b026124e75cec065f8920ffb2a"
server_key_version: "encryptionkey@0"
I20260926 02:57:16.314553 4401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:16.344488 4401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:16.349222 4401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:16.349591 4401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:16.350693 4401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:16.352425 4401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:16.352568 4401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:16.352720 4401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:16.352831 4401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:16.389637 4401 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:46763
I20260926 02:57:16.389694 4568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:46763 every 8 connection(s)
I20260926 02:57:16.391225 4401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:57:16.393416 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 4401
I20260926 02:57:16.394177 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:34651
--local_ip_for_outbound_sockets=127.19.185.194
--tserver_master_addrs=127.19.185.254:38225
--webserver_port=41671
--webserver_interface=127.19.185.194
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:57:16.397368 4571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:16.394982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:40987)
0926 02:57:16.395877 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:16.396202 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":355,"threads_started":1}
W20260926 02:57:16.398257 4569 heartbeater.cc:644] Failed to heartbeat to 127.19.185.254:38225 (0 consecutive failures): Network error: Failed to ping master at 127.19.185.254:38225: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
I20260926 02:57:16.399394 4571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:16.398910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:60469)
0926 02:57:16.399107 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:16.399203 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 02:57:16.400425 4571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:16.400003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:39201)
0926 02:57:16.400169 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:16.400265 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:57:16.400710 4569 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:57:16.737602 4572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:16.738025 4572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:16.738209 4572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:16.745836 4572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:16.746047 4572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:16.746253 4572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:57:16.756891 4572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:34651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=41671
--tserver_master_addrs=127.19.185.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:16.758313 4572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:16.759837 4572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:16.773941 4577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:16.773828 4580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:16.773876 4578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:16.775081 4572 server_base.cc:1034] running on GCE node
I20260926 02:57:16.775792 4572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:16.776784 4572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:16.778035 4572 hybrid_clock.cc:648] HybridClock initialized: now 1790391436778004 us; error 51 us; skew 500 ppm
I20260926 02:57:16.778447 4572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:16 dist-test-slave-ktng krb5kdc[3252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391436, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:16.781934 4572 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:57:16.783561 4572 webserver.cc:460] Webserver started at http://127.19.185.194:41671/ using document root <none> and password file <none>
I20260926 02:57:16.784277 4572 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:16.784423 4572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:16.789789 4572 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 02:57:16.792716 4587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:16.794215 4572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:57:16.794406 4572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "94d5768070d84037a35f0ba378c5e3f8"
format_stamp: "Formatted at 2026-09-26 02:56:31 on dist-test-slave-ktng"
server_key: "e68ca22310fe5fbf5278953256ca82ed"
server_key_iv: "c436552ef6ea009dd05a32ddea31f919"
server_key_version: "encryptionkey@0"
I20260926 02:57:16.795413 4572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:16.821452 4572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:16.826061 4572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:16.826380 4572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:16.827611 4572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:16.829089 4572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:16.829224 4572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:16.829361 4572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:16.829438 4572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:16.866099 4572 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:34651
I20260926 02:57:16.866194 4699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:34651 every 8 connection(s)
I20260926 02:57:16.867729 4572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:57:16.870332 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 4572
I20260926 02:57:16.870993 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:35819
--local_ip_for_outbound_sockets=127.19.185.195
--tserver_master_addrs=127.19.185.254:38225
--webserver_port=32891
--webserver_interface=127.19.185.195
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:57:16.873032 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:16.870894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:42503)
0926 02:57:16.871830 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:16.872132 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":445,"threads_started":1}
W20260926 02:57:16.873739 4700 heartbeater.cc:644] Failed to heartbeat to 127.19.185.254:38225 (0 consecutive failures): Network error: Failed to ping master at 127.19.185.254:38225: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
I20260926 02:57:16.874678 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:16.874330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:46241)
0926 02:57:16.874495 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:16.874567 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 02:57:16.875562 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:16.875203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:46691)
0926 02:57:16.875367 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:16.875454 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:57:16.875772 4700 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:57:17.202809 4703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:17.203080 4703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:17.203279 4703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:17.210537 4703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:17.210633 4703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:17.210830 4703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:57:17.220510 4703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:35819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=32891
--tserver_master_addrs=127.19.185.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:17.221755 4703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:17.223073 4703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:17.236099 4711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:17.236174 4709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:17.236289 4708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:17.237031 4703 server_base.cc:1034] running on GCE node
I20260926 02:57:17.237625 4703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:17.238572 4703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:17.239770 4703 hybrid_clock.cc:648] HybridClock initialized: now 1790391437239721 us; error 44 us; skew 500 ppm
I20260926 02:57:17.240103 4703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:17 dist-test-slave-ktng krb5kdc[3252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391437, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:17.243531 4703 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:57:17.245019 4703 webserver.cc:460] Webserver started at http://127.19.185.195:32891/ using document root <none> and password file <none>
I20260926 02:57:17.245721 4703 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:17.245868 4703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:17.251122 4703 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:57:17.254160 4718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:17.255616 4703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:57:17.255793 4703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "9c26acfcdf844eb7b8476d1aa2018f76"
format_stamp: "Formatted at 2026-09-26 02:56:31 on dist-test-slave-ktng"
server_key: "60d0ab52cafc7c2b8bf33cc1d391ecf5"
server_key_iv: "565747288b75dac7d8801f55ec9728e3"
server_key_version: "encryptionkey@0"
I20260926 02:57:17.256767 4703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:17.294552 4703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:17.298988 4703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:17.299273 4703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:17.300450 4703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:17.301826 4703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:17.301949 4703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:17.302078 4703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:17.302175 4703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:17.335341 4703 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:35819
I20260926 02:57:17.335443 4830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:35819 every 8 connection(s)
I20260926 02:57:17.336874 4703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:57:17.341872 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:17.339969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:34241)
0926 02:57:17.340721 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:17.340975 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":279,"threads_started":1}
W20260926 02:57:17.342557 4831 heartbeater.cc:644] Failed to heartbeat to 127.19.185.254:38225 (0 consecutive failures): Network error: Failed to ping master at 127.19.185.254:38225: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
I20260926 02:57:17.343397 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:17.343027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:48471)
0926 02:57:17.343173 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:17.343239 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 02:57:17.344242 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:17.343919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:46157)
0926 02:57:17.344058 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:17.344138 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 02:57:17.344563 4831 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 02:57:17.346342 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 4703
I20260926 02:57:17.402062 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:17.401266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:41301)
0926 02:57:17.401734 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:17.401825 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":225,"threads_started":1}
I20260926 02:57:17.877535 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:17.876457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:54245)
0926 02:57:17.876998 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:17.877155 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":253,"threads_started":1}
I20260926 02:57:18.346024 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:18.345172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:38599)
0926 02:57:18.345677 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:18.345797 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":263,"threads_started":1}
I20260926 02:57:18.403774 4838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:18.402981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:44681)
0926 02:57:18.403520 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:18.403606 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":239,"threads_started":1}
I20260926 02:57:18.879482 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:18.878542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:41541)
0926 02:57:18.879047 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:18.879139 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":234,"threads_started":1}
I20260926 02:57:19.347893 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:19.346971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:36291)
0926 02:57:19.347519 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:19.347622 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":247,"threads_started":1}
I20260926 02:57:19.405392 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:19.404688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:43347)
0926 02:57:19.405150 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:19.405227 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":252,"threads_started":1}
I20260926 02:57:19.881073 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:19.880330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:34699)
0926 02:57:19.880791 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:19.880883 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":247,"threads_started":1}
I20260926 02:57:20.349602 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:20.348872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:45039)
0926 02:57:20.349324 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:20.349432 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":244,"threads_started":1}
I20260926 02:57:20.407347 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:20.406360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:55007)
0926 02:57:20.406977 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:20.407061 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":272,"threads_started":1}
I20260926 02:57:20.882999 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:20.882170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:59589)
0926 02:57:20.882705 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:20.882793 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":283,"threads_started":1}
I20260926 02:57:21.351487 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:21.350672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:48563)
0926 02:57:21.351123 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:21.351203 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":244,"threads_started":1}
I20260926 02:57:21.408943 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:21.408225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:44291)
0926 02:57:21.408667 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:21.408779 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":237,"threads_started":1}
I20260926 02:57:21.884852 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:21.884127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:35499)
0926 02:57:21.884596 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:21.884680 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":227,"threads_started":1}
I20260926 02:57:22.353327 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:22.352495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:60675)
0926 02:57:22.353075 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:22.353157 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":312,"threads_started":1}
I20260926 02:57:22.410477 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:22.409775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:60403)
0926 02:57:22.410202 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:22.410290 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":218,"threads_started":1}
I20260926 02:57:22.886631 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:22.885907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:50037)
0926 02:57:22.886381 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:22.886468 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":246,"threads_started":1}
I20260926 02:57:23.354951 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:23.354235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:57195)
0926 02:57:23.354687 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:23.354776 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":239,"threads_started":1}
I20260926 02:57:23.412261 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:23.411444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:53649)
0926 02:57:23.411967 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:23.412053 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":207,"threads_started":1}
I20260926 02:57:23.888484 4854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:23.887604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:45837)
0926 02:57:23.888148 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:23.888266 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":284,"threads_started":1}
I20260926 02:57:24.356662 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:24.355886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:44019)
0926 02:57:24.356364 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:24.356452 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":221,"threads_started":1}
I20260926 02:57:24.413939 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:24.413273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:48755)
0926 02:57:24.413700 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:24.413781 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":216,"threads_started":1}
I20260926 02:57:24.890306 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:24.889563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:54813)
0926 02:57:24.890060 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:24.890143 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":239,"threads_started":1}
I20260926 02:57:25.358433 4858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:25.357721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:60031)
0926 02:57:25.358180 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:25.358263 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":251,"threads_started":1}
I20260926 02:57:25.415771 4859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:25.414938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:39301)
0926 02:57:25.415485 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:25.415598 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":246,"threads_started":1}
I20260926 02:57:25.892088 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:25.891280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:49987)
0926 02:57:25.891807 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:25.891909 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":236,"threads_started":1}
I20260926 02:57:26.360106 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:26.359423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:44521)
0926 02:57:26.359852 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:26.359939 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":206,"threads_started":1}
I20260926 02:57:26.417575 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:26.416804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:33045)
0926 02:57:26.417303 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:26.417414 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":271,"threads_started":1}
I20260926 02:57:26.893874 4863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:26.893123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:37317)
0926 02:57:26.893629 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:26.893711 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":287,"threads_started":1}
I20260926 02:57:27.361805 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:27.361050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:43947)
0926 02:57:27.361557 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:27.361638 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":310,"threads_started":1}
I20260926 02:57:27.419266 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:27.418517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:37347)
0926 02:57:27.418948 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:27.419039 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":218,"threads_started":1}
I20260926 02:57:27.895919 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:27.895016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:49685)
0926 02:57:27.895577 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:27.895687 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":311,"threads_started":1}
I20260926 02:57:28.363597 4867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:28.362715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:37259)
0926 02:57:28.363157 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:28.363249 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":238,"threads_started":1}
I20260926 02:57:28.421029 4868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:28.420324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:41267)
0926 02:57:28.420777 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:28.420869 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":246,"threads_started":1}
I20260926 02:57:28.897588 4869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:28.896868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:41331)
0926 02:57:28.897313 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:28.897425 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":211,"threads_started":1}
I20260926 02:57:29.365142 4870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:29.364465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:51745)
0926 02:57:29.364899 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:29.364980 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":220,"threads_started":1}
I20260926 02:57:29.422709 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:29.421942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:59081)
0926 02:57:29.422419 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:29.422531 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":253,"threads_started":1}
I20260926 02:57:29.899652 4872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:29.898673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:49307)
0926 02:57:29.899358 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:29.899463 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":358,"threads_started":1}
I20260926 02:57:30.366905 4873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:30.366090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:43681)
0926 02:57:30.366624 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:30.366721 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":290,"threads_started":1}
I20260926 02:57:30.424368 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:30.423647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:40269)
0926 02:57:30.424097 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:30.424178 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":233,"threads_started":1}
I20260926 02:57:30.901315 4875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:30.900548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:42859)
0926 02:57:30.901027 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:30.901126 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260926 02:57:31.368530 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:31.367814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:58601)
0926 02:57:31.368264 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:31.368343 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":226,"threads_started":1}
I20260926 02:57:31.426350 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:31.425589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.193:58533)
0926 02:57:31.426077 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:31.426174 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":266,"threads_started":1}
I20260926 02:57:31.903275 4878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:31.902472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.194:52545)
0926 02:57:31.902972 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:31.903092 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":294,"threads_started":1}
I20260926 02:57:32.352280 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 4401
I20260926 02:57:32.370307 4880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:32.369549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38225 (local address 127.19.185.195:47055)
0926 02:57:32.370038 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:32.370133 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.254:38225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":259,"threads_started":1}
I20260926 02:57:32.372008 4879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:32.353924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46076 (local address 127.19.185.193:46763)
0926 02:57:32.354566 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:32.354581 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:32.354621 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:57:32.356212 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:32.356222 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:32.356693 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:32.356996 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:32.358567 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:32.359930 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:32.361518 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:32.362273 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:32.362408 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:32.365452 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:32.365460 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:32.365476 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:32.365518 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:32.368528 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:32.369231 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:32.369238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:32.369245 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:32.369419 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:32.369811 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:32.369819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:32.369822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:32.370349 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:32.370683 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:32.371202 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 02:57:32.371694 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":310,"threads_started":1}
I20260926 02:57:32.373421 4564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:32.460513 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 4401
I20260926 02:57:32.475445 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 4572
I20260926 02:57:32.494238 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:32.476374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.19.185.194:34651)
0926 02:57:32.476956 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:32.476972 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:32.477010 (+ 38us) server_negotiation.cc:407] Connection header received
0926 02:57:32.478522 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:32.478532 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:32.478992 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:32.479398 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:32.480757 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:32.482046 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:32.483403 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:32.484153 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:32.484320 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:32.487071 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:32.487078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:32.487094 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:32.487138 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:32.490262 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:32.491107 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:32.491114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:32.491121 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:32.491632 (+ 511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:32.492162 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:32.492166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:32.492168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:32.492730 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:32.492989 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:32.493487 (+ 498us) server_negotiation.cc:299] Negotiation successful
0926 02:57:32.493827 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":264,"threads_started":1}
I20260926 02:57:32.495704 4695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:32.581984 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 4572
I20260926 02:57:32.599574 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 4703
I20260926 02:57:32.618266 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:32.600645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38068 (local address 127.19.185.195:35819)
0926 02:57:32.601295 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:32.601309 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:32.601352 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:57:32.603066 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:32.603076 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:32.603612 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:32.603920 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:32.605306 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:32.606582 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:32.608035 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:32.608782 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:32.608930 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:32.611516 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:32.611523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:32.611540 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:32.611577 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:32.614812 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:32.615591 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:32.615599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:32.615606 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:32.615849 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:32.616191 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:32.616194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:32.616196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:32.616775 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:32.617016 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:32.617421 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 02:57:32.617958 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":309,"threads_started":1}
I20260926 02:57:32.619972 4826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:32.706895 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 4703
2026-09-26T02:57:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64622 ms)
[----------] 30 tests from SecurityITest (301125 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:57:32 dist-test-slave-ktng krb5kdc[4889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:57:32 dist-test-slave-ktng krb5kdc[4889](info): set up 2 sockets
Sep 26 02:57:32 dist-test-slave-ktng krb5kdc[4889](info): commencing operation
krb5kdc: starting...
W20260926 02:57:33.816313 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:57:33 dist-test-slave-ktng krb5kdc[4889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:57:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:57:34.079926 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:44853
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:44853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:34.417249 4902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:34.417541 4902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:34.417714 4902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:34.425004 4902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:57:34.425108 4902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:34.425213 4902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:34.425319 4902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:57:34.425391 4902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:57:34.435400 4902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:44853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:44853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:34.436728 4902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:34.438091 4902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:34.451264 4907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:34.451344 4908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:34.451319 4910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:34.451726 4902 server_base.cc:1034] running on GCE node
I20260926 02:57:34.452747 4902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:34.454201 4902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:34.455423 4902 hybrid_clock.cc:648] HybridClock initialized: now 1790391454455353 us; error 44 us; skew 500 ppm
I20260926 02:57:34.455772 4902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:34 dist-test-slave-ktng krb5kdc[4889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391454, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:34.458973 4902 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:57:34.460626 4902 webserver.cc:460] Webserver started at http://127.19.185.254:34613/ using document root <none> and password file <none>
I20260926 02:57:34.461321 4902 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:34.461472 4902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:34.461778 4902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:34.464907 4902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "8b57a5b5c38847b8a6255d31afc2f63d"
format_stamp: "Formatted at 2026-09-26 02:57:34 on dist-test-slave-ktng"
server_key: "3a540897c0575dac54ca9f1c32bf8c2f"
server_key_iv: "32b9ef0a0a1cfd55cdb63e805c946c6f"
server_key_version: "encryptionkey@0"
I20260926 02:57:34.465672 4902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "8b57a5b5c38847b8a6255d31afc2f63d"
format_stamp: "Formatted at 2026-09-26 02:57:34 on dist-test-slave-ktng"
server_key: "3a540897c0575dac54ca9f1c32bf8c2f"
server_key_iv: "32b9ef0a0a1cfd55cdb63e805c946c6f"
server_key_version: "encryptionkey@0"
I20260926 02:57:34.470824 4902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:57:34.474252 4917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:34.475610 4902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:57:34.475800 4902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "8b57a5b5c38847b8a6255d31afc2f63d"
format_stamp: "Formatted at 2026-09-26 02:57:34 on dist-test-slave-ktng"
server_key: "3a540897c0575dac54ca9f1c32bf8c2f"
server_key_iv: "32b9ef0a0a1cfd55cdb63e805c946c6f"
server_key_version: "encryptionkey@0"
I20260926 02:57:34.476007 4902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:34.503130 4902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:34.507581 4902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:34.507879 4902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:34.527143 4902 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:44853
I20260926 02:57:34.527180 4968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:44853 every 8 connection(s)
I20260926 02:57:34.528625 4902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:57:34.532521 4969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:34.535072 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 4902
I20260926 02:57:34.535316 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:57:34.536098 20199 external_mini_cluster.cc:1428] Setting key 107e22bdea7d77867ee0b5361895a605
I20260926 02:57:34.545466 4969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d: Bootstrap starting.
I20260926 02:57:34.549335 4969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:34.550683 4969 log.cc:824] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:57:34 dist-test-slave-ktng krb5kdc[4889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:57:34.554157 4969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d: No bootstrap required, opened a new log
I20260926 02:57:34.560706 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:34.538285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.19.185.254:44853)
0926 02:57:34.539104 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:34.539116 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:34.539162 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:57:34.540904 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:34.540927 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:34.541307 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:34.541666 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:34.543501 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:34.544784 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:34.546643 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:34.547640 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:34.547774 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:34.553203 (+ 5429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:34.553211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:34.553225 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:34.553258 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:34.556595 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:34.557436 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:34.557442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:34.557448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:34.557617 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:34.558035 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:34.558039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:34.558041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:34.558529 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:34.558740 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:34.559201 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 02:57:34.559677 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":386,"threads_started":1}
I20260926 02:57:34.562991 4969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b57a5b5c38847b8a6255d31afc2f63d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44853 } }
I20260926 02:57:34.563392 4969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:34.563524 4969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b57a5b5c38847b8a6255d31afc2f63d, State: Initialized, Role: FOLLOWER
I20260926 02:57:34.564055 4969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b57a5b5c38847b8a6255d31afc2f63d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44853 } }
I20260926 02:57:34.564314 4969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:57:34.564458 4969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:57:34.564633 4969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:34.566848 4969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b57a5b5c38847b8a6255d31afc2f63d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44853 } }
I20260926 02:57:34.567402 4969 leader_election.cc:302] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b57a5b5c38847b8a6255d31afc2f63d; no voters:
I20260926 02:57:34.568027 4969 leader_election.cc:288] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:57:34.568161 4974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:57:34.569176 4974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [term 1 LEADER]: Becoming Leader. State: Replica: 8b57a5b5c38847b8a6255d31afc2f63d, State: Running, Role: LEADER
I20260926 02:57:34.569984 4974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b57a5b5c38847b8a6255d31afc2f63d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44853 } }
I20260926 02:57:34.570199 4969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:57:34.573163 4975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b57a5b5c38847b8a6255d31afc2f63d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b57a5b5c38847b8a6255d31afc2f63d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44853 } } }
I20260926 02:57:34.573113 4976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b57a5b5c38847b8a6255d31afc2f63d. Latest consensus state: current_term: 1 leader_uuid: "8b57a5b5c38847b8a6255d31afc2f63d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b57a5b5c38847b8a6255d31afc2f63d" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44853 } } }
I20260926 02:57:34.573589 4975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:34.573621 4976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:34.580672 4983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:57:34.585960 4983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:57:34.595031 4983 catalog_manager.cc:1347] Generated new cluster ID: 72c32b71799c464aa64e4f903caf4faa
I20260926 02:57:34.595165 4983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:57:34.618196 4983 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:57:34.620129 4983 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:57:34.634528 4983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d: Generated new TSK 0
I20260926 02:57:34.635751 4983 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:57:34.703511 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:44853
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:35.041374 4995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:35.041643 4995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:35.041800 4995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:35.049237 4995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:35.049341 4995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:35.049515 4995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:57:35.059393 4995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:35.060683 4995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:35.062027 4995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:35.076018 5000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:35.076072 5003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:35.076107 5001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:35.077394 4995 server_base.cc:1034] running on GCE node
I20260926 02:57:35.078048 4995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:35.079098 4995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:35.080335 4995 hybrid_clock.cc:648] HybridClock initialized: now 1790391455080229 us; error 98 us; skew 500 ppm
I20260926 02:57:35.080703 4995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:35 dist-test-slave-ktng krb5kdc[4889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391455, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:35.089458 4995 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:57:35.091270 4995 webserver.cc:460] Webserver started at http://127.19.185.193:42383/ using document root <none> and password file <none>
I20260926 02:57:35.092131 4995 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:35.092260 4995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:35.092592 4995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:35.095667 4995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "ac190177f1ca42f99e9ed857b3e54470"
format_stamp: "Formatted at 2026-09-26 02:57:35 on dist-test-slave-ktng"
server_key: "652ec92087e573c1a952899cda415d7c"
server_key_iv: "f8ff73431502604946b10f06a3984e24"
server_key_version: "encryptionkey@0"
I20260926 02:57:35.096444 4995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "ac190177f1ca42f99e9ed857b3e54470"
format_stamp: "Formatted at 2026-09-26 02:57:35 on dist-test-slave-ktng"
server_key: "652ec92087e573c1a952899cda415d7c"
server_key_iv: "f8ff73431502604946b10f06a3984e24"
server_key_version: "encryptionkey@0"
I20260926 02:57:35.101752 4995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:57:35.105393 5010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:35.106736 4995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:57:35.106918 4995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "ac190177f1ca42f99e9ed857b3e54470"
format_stamp: "Formatted at 2026-09-26 02:57:35 on dist-test-slave-ktng"
server_key: "652ec92087e573c1a952899cda415d7c"
server_key_iv: "f8ff73431502604946b10f06a3984e24"
server_key_version: "encryptionkey@0"
I20260926 02:57:35.107127 4995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:35.128566 4995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:35.133150 4995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:35.133455 4995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:35.134547 4995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:35.135968 4995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:35.136072 4995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:35.136234 4995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:35.136310 4995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:35.173724 4995 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:45255
I20260926 02:57:35.173776 5122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:45255 every 8 connection(s)
I20260926 02:57:35.175128 4995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:57:35.182045 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 4995
I20260926 02:57:35.182286 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:57:35.183122 20199 external_mini_cluster.cc:1428] Setting key 4f04e30aadcf59eb8378a3b6f06b7756
Sep 26 02:57:35 dist-test-slave-ktng krb5kdc[4889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391455, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:57:35.202200 5125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:35.177925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:40221 (local address 127.19.185.254:44853)
0926 02:57:35.178420 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:35.178426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:35.179533 (+ 1107us) server_negotiation.cc:407] Connection header received
0926 02:57:35.181867 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:35.181871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:35.181944 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:35.182041 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:35.184129 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:35.185062 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:35.189820 (+ 4758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:35.190229 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:35.190343 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:35.194694 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:35.194702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:35.194710 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:35.194758 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:35.198625 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:35.199395 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:35.199403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:35.199410 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:35.199550 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:35.200086 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:35.200094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:35.200100 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:35.200517 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:35.200668 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:35.201455 (+ 787us) server_negotiation.cc:299] Negotiation successful
0926 02:57:35.201715 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":234,"threads_started":1}
I20260926 02:57:35.202797 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:35.178503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:44853 (local address 127.19.185.193:40221)
0926 02:57:35.179287 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:35.179386 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:35.181452 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:35.182358 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:35.182372 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:35.183169 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:35.183973 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:35.183994 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:35.187448 (+ 3454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:35.187457 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:35.189625 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:35.189643 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:35.190477 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:35.190486 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:35.190851 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:35.191673 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:57:35.191713 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:35.194427 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:35.198775 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:35.198782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:35.198796 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:35.199222 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:35.199657 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:35.199660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:35.199662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:35.199958 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:35.200652 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:35.200658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:35.201044 (+ 386us) client_negotiation.cc:769] Sending connection context
0926 02:57:35.201280 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 02:57:35.201660 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":276,"threads_started":1}
I20260926 02:57:35.204517 5123 heartbeater.cc:347] Connected to a master server at 127.19.185.254:44853
I20260926 02:57:35.204843 5123 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:35.205572 5123 heartbeater.cc:511] Master 127.19.185.254:44853 requested a full tablet report, sending...
I20260926 02:57:35.207746 4934 ts_manager.cc:194] Registered new tserver with Master: ac190177f1ca42f99e9ed857b3e54470 (127.19.185.193:45255)
I20260926 02:57:35.209200 4934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:40221
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:57:35.242458 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:44853
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:35.574599 5129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:35.574888 5129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:35.574987 5129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:35.582540 5129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:35.582654 5129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:35.582767 5129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:57:35.592664 5129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:35.593916 5129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:35.595176 5129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:35.608342 5135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:35.608592 5134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:35.608572 5137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:35.609444 5129 server_base.cc:1034] running on GCE node
I20260926 02:57:35.610046 5129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:35.611052 5129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:35.612323 5129 hybrid_clock.cc:648] HybridClock initialized: now 1790391455612191 us; error 113 us; skew 500 ppm
I20260926 02:57:35.612669 5129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:35 dist-test-slave-ktng krb5kdc[4889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391455, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:35.616225 5129 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:57:35.617806 5129 webserver.cc:460] Webserver started at http://127.19.185.194:43633/ using document root <none> and password file <none>
I20260926 02:57:35.618494 5129 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:35.618623 5129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:35.618955 5129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:35.622001 5129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "acc369db30aa4c3bbe99c2051e8d6070"
format_stamp: "Formatted at 2026-09-26 02:57:35 on dist-test-slave-ktng"
server_key: "6f8c8abe70f64cbeb7f7a374224a65cf"
server_key_iv: "9378bf390f09a33bf17312879d5404d3"
server_key_version: "encryptionkey@0"
I20260926 02:57:35.622809 5129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "acc369db30aa4c3bbe99c2051e8d6070"
format_stamp: "Formatted at 2026-09-26 02:57:35 on dist-test-slave-ktng"
server_key: "6f8c8abe70f64cbeb7f7a374224a65cf"
server_key_iv: "9378bf390f09a33bf17312879d5404d3"
server_key_version: "encryptionkey@0"
I20260926 02:57:35.628464 5129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:57:35.632447 5144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:35.634074 5129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260926 02:57:35.634265 5129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "acc369db30aa4c3bbe99c2051e8d6070"
format_stamp: "Formatted at 2026-09-26 02:57:35 on dist-test-slave-ktng"
server_key: "6f8c8abe70f64cbeb7f7a374224a65cf"
server_key_iv: "9378bf390f09a33bf17312879d5404d3"
server_key_version: "encryptionkey@0"
I20260926 02:57:35.634495 5129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:35.658248 5129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:35.662765 5129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:35.663076 5129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:35.664215 5129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:35.665578 5129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:35.665690 5129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:35.665841 5129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:35.665915 5129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:35.700093 5129 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:42669
I20260926 02:57:35.700120 5256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:42669 every 8 connection(s)
I20260926 02:57:35.701399 5129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:57:35.708108 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 5129
I20260926 02:57:35.708331 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:57:35.708892 20199 external_mini_cluster.cc:1428] Setting key 45a6a0945adc66949ddd895e08604fe5
Sep 26 02:57:35 dist-test-slave-ktng krb5kdc[4889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391455, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:57:35.723064 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:35.704221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:35507 (local address 127.19.185.254:44853)
0926 02:57:35.704680 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:35.704686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:35.705853 (+ 1167us) server_negotiation.cc:407] Connection header received
0926 02:57:35.708020 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:35.708025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:35.708100 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:35.708248 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:35.710274 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:35.711253 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:35.712949 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:35.713398 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:35.713532 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:35.717080 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:35.717085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:35.717089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:35.717122 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:35.719965 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:35.720711 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:35.720721 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:35.720723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:35.720851 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:35.721294 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:35.721299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:35.721306 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:35.721569 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:35.721724 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:35.722418 (+ 694us) server_negotiation.cc:299] Negotiation successful
0926 02:57:35.722666 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":212,"threads_started":1}
I20260926 02:57:35.724108 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:35.704926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:44853 (local address 127.19.185.194:35507)
0926 02:57:35.705656 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:35.705712 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:35.707614 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:35.708534 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:35.708547 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:35.709337 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:35.710086 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:35.710107 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:35.711457 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:35.711461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:35.712796 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:35.712809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:35.713644 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:35.713651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:35.713887 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:35.714585 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:57:35.714613 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:35.716830 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:35.720115 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:35.720125 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:35.720143 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:35.720574 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:35.720966 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:35.720971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:35.720974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:35.721172 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:35.721697 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:35.721706 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:35.722068 (+ 362us) client_negotiation.cc:769] Sending connection context
0926 02:57:35.722342 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 02:57:35.722910 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":293,"threads_started":1}
I20260926 02:57:35.725677 5257 heartbeater.cc:347] Connected to a master server at 127.19.185.254:44853
I20260926 02:57:35.726024 5257 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:35.726934 5257 heartbeater.cc:511] Master 127.19.185.254:44853 requested a full tablet report, sending...
I20260926 02:57:35.728658 4934 ts_manager.cc:194] Registered new tserver with Master: acc369db30aa4c3bbe99c2051e8d6070 (127.19.185.194:42669)
I20260926 02:57:35.729967 4934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:35507
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:57:35.768267 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:44853
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:36.097208 5263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:36.097487 5263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:36.097648 5263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:36.104929 5263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:36.105032 5263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:36.105202 5263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:57:36.114902 5263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:36.116261 5263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:36.117591 5263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:36.131776 5268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:36.131867 5271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:36.131841 5269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:36.132710 5263 server_base.cc:1034] running on GCE node
I20260926 02:57:36.133343 5263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:36.134438 5263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:36.135649 5263 hybrid_clock.cc:648] HybridClock initialized: now 1790391456135592 us; error 47 us; skew 500 ppm
I20260926 02:57:36.136009 5263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:36 dist-test-slave-ktng krb5kdc[4889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391456, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:36.139122 5263 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:57:36.140617 5263 webserver.cc:460] Webserver started at http://127.19.185.195:43757/ using document root <none> and password file <none>
I20260926 02:57:36.141327 5263 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:36.141475 5263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:36.141819 5263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:36.144814 5263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "5b703f039cec44db97083aca3b265a05"
format_stamp: "Formatted at 2026-09-26 02:57:36 on dist-test-slave-ktng"
server_key: "617e5168a978e911fe32b362496a9c1b"
server_key_iv: "bf43452d97e911d23f482de2bf3ccbde"
server_key_version: "encryptionkey@0"
I20260926 02:57:36.145576 5263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "5b703f039cec44db97083aca3b265a05"
format_stamp: "Formatted at 2026-09-26 02:57:36 on dist-test-slave-ktng"
server_key: "617e5168a978e911fe32b362496a9c1b"
server_key_iv: "bf43452d97e911d23f482de2bf3ccbde"
server_key_version: "encryptionkey@0"
I20260926 02:57:36.150728 5263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:57:36.154232 5278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:36.155588 5263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:57:36.155752 5263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "5b703f039cec44db97083aca3b265a05"
format_stamp: "Formatted at 2026-09-26 02:57:36 on dist-test-slave-ktng"
server_key: "617e5168a978e911fe32b362496a9c1b"
server_key_iv: "bf43452d97e911d23f482de2bf3ccbde"
server_key_version: "encryptionkey@0"
I20260926 02:57:36.156006 5263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:36.172679 5263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:36.177765 5263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:36.178098 5263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:36.179237 5263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:36.180917 5263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:36.181028 5263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:36.181200 5263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:36.181282 5263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:36.212450 5123 heartbeater.cc:503] Master 127.19.185.254:44853 was elected leader, sending a full tablet report...
I20260926 02:57:36.220713 5263 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:45841
I20260926 02:57:36.220764 5390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:45841 every 8 connection(s)
I20260926 02:57:36.222110 5263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:57:36.223654 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 5263
I20260926 02:57:36.223873 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:57:36.224341 20199 external_mini_cluster.cc:1428] Setting key 4b547b428352c33bd41899486340b631
Sep 26 02:57:36 dist-test-slave-ktng krb5kdc[4889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391456, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:57:36.244467 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.225401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:41941 (local address 127.19.185.254:44853)
0926 02:57:36.225940 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:36.225945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:36.226949 (+ 1004us) server_negotiation.cc:407] Connection header received
0926 02:57:36.228859 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:36.228863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:36.228943 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:36.229038 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:36.230642 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.231608 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.233348 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.233768 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.233870 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.237787 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:36.237795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:36.237798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:36.237833 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:36.241451 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:36.242230 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:36.242234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:36.242236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:36.242369 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:36.242807 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:36.242810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:36.242812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:36.243073 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:36.243194 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:36.243998 (+ 804us) server_negotiation.cc:299] Negotiation successful
0926 02:57:36.244174 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":281,"threads_started":1}
I20260926 02:57:36.245437 5394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.225937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:44853 (local address 127.19.185.195:41941)
0926 02:57:36.226803 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:36.226850 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:36.228609 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:36.229248 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:36.229255 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:36.229884 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:36.230461 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:36.230479 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.231835 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.231840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:36.233201 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:36.233214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.233973 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.233979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:36.234192 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.235111 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:57:36.235135 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:36.237573 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:36.241593 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:36.241602 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:36.241617 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:36.242101 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:36.242501 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:36.242505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:36.242507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:36.242703 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:36.243240 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:36.243247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:36.243660 (+ 413us) client_negotiation.cc:769] Sending connection context
0926 02:57:36.243895 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 02:57:36.244268 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":353,"threads_started":1}
I20260926 02:57:36.246850 5391 heartbeater.cc:347] Connected to a master server at 127.19.185.254:44853
I20260926 02:57:36.247157 5391 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:36.247973 5391 heartbeater.cc:511] Master 127.19.185.254:44853 requested a full tablet report, sending...
I20260926 02:57:36.249379 4934 ts_manager.cc:194] Registered new tserver with Master: 5b703f039cec44db97083aca3b265a05 (127.19.185.195:45841)
I20260926 02:57:36.250206 4934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:41941
I20260926 02:57:36.253429 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:57:36.272014 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.258020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54936 (local address 127.19.185.254:44853)
0926 02:57:36.258187 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:36.258191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:36.258702 (+ 511us) server_negotiation.cc:407] Connection header received
0926 02:57:36.258826 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:36.258830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:36.258915 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:36.259052 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:36.260734 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.261457 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.262784 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.263255 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.263447 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.266274 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:36.266280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:36.266283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:36.266313 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:36.268941 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:36.269653 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:36.269657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:36.269660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:36.269885 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:36.270311 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:36.270320 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:36.270327 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:36.270668 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:36.271025 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:36.271429 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 02:57:36.271666 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:57:36.276234 4934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:57:36.283975 4934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:57:36.319561 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.298408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.19.185.193:45255)
0926 02:57:36.309614 (+ 11206us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:36.309626 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:36.310899 (+ 1273us) server_negotiation.cc:407] Connection header received
0926 02:57:36.311057 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:36.311069 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:36.311453 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:36.311678 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:36.312366 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.313940 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.316793 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.318063 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.318262 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.318522 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:36.318983 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 02:57:36.319173 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10963,"thread_start_us":10764,"threads_started":1}
I20260926 02:57:36.319672 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.297943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:45255 (local address 127.0.0.1:55578)
0926 02:57:36.309953 (+ 12010us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:36.309972 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:36.310106 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:36.311684 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:36.311694 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:36.311724 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:36.312170 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:36.312185 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.314129 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.314134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:36.316632 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:36.316645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.318495 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.318507 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:36.318717 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.318773 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 02:57:36.318937 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 02:57:36.319147 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11870,"mutex_wait_us":1,"thread_start_us":11715,"threads_started":1}
I20260926 02:57:36.321779 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.297351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:45841 (local address 127.0.0.1:38650)
0926 02:57:36.304568 (+ 7217us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:36.304617 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:36.304819 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:36.312996 (+ 8177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:36.313001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:36.313025 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:36.314529 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:36.314548 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.316447 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.316453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:36.318953 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:36.318967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.320785 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.320797 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:36.321003 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.321057 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 02:57:36.321156 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 02:57:36.321349 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6951,"mutex_wait_us":1,"thread_start_us":6782,"threads_started":1}
I20260926 02:57:36.321931 5405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.297435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.19.185.195:45841)
0926 02:57:36.312385 (+ 14950us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:36.312393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:36.312440 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:57:36.312542 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:36.312549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:36.312780 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:36.314444 (+ 1664us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:36.314698 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.316297 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.319129 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.320557 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.320749 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.320917 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:36.321232 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 02:57:36.321399 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14757,"thread_start_us":16641,"threads_started":1}
I20260926 02:57:36.324499 5409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.309728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:42669 (local address 127.0.0.1:42512)
0926 02:57:36.314159 (+ 4431us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:36.314178 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:36.314347 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:36.315060 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:36.315065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:36.315095 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:36.315622 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:36.315634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.317681 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.317705 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:36.320953 (+ 3248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:36.320969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.322920 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.322928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:36.323115 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.323916 (+ 801us) client_negotiation.cc:769] Sending connection context
0926 02:57:36.323990 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 02:57:36.324126 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4297,"mutex_wait_us":1071,"thread_start_us":4122,"threads_started":1}
I20260926 02:57:36.324193 5058 tablet_service.cc:1467] Processing CreateTablet for tablet bfa46ae419954f3684a9dd73a3e8abd6 (DEFAULT_TABLE table=test-table [id=87686dc8ec8b409dbfdbc7e61460d822]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:36.325310 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.305391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.19.185.194:42669)
0926 02:57:36.313802 (+ 8411us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:36.313813 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:36.314522 (+ 709us) server_negotiation.cc:407] Connection header received
0926 02:57:36.314642 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:36.314652 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:36.314851 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:36.315060 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:36.315905 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.317477 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.321134 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.322680 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.322860 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.323009 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:36.324845 (+ 1836us) server_negotiation.cc:299] Negotiation successful
0926 02:57:36.325014 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8221,"thread_start_us":8109,"threads_started":1}
I20260926 02:57:36.326107 5058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfa46ae419954f3684a9dd73a3e8abd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:36.326087 5326 tablet_service.cc:1467] Processing CreateTablet for tablet bfa46ae419954f3684a9dd73a3e8abd6 (DEFAULT_TABLE table=test-table [id=87686dc8ec8b409dbfdbc7e61460d822]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:36.327744 5326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfa46ae419954f3684a9dd73a3e8abd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:36.329113 5192 tablet_service.cc:1467] Processing CreateTablet for tablet bfa46ae419954f3684a9dd73a3e8abd6 (DEFAULT_TABLE table=test-table [id=87686dc8ec8b409dbfdbc7e61460d822]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:36.330590 5192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfa46ae419954f3684a9dd73a3e8abd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:36.337663 5410 tablet_bootstrap.cc:492] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470: Bootstrap starting.
I20260926 02:57:36.341657 5410 tablet_bootstrap.cc:654] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:36.342674 5410 log.cc:824] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:36.343981 5411 tablet_bootstrap.cc:492] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05: Bootstrap starting.
I20260926 02:57:36.346951 5413 tablet_bootstrap.cc:492] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070: Bootstrap starting.
I20260926 02:57:36.347493 5410 tablet_bootstrap.cc:492] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470: No bootstrap required, opened a new log
I20260926 02:57:36.347831 5410 ts_tablet_manager.cc:1397] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260926 02:57:36.347945 5411 tablet_bootstrap.cc:654] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:36.349680 5411 log.cc:824] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:36.350219 5413 tablet_bootstrap.cc:654] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:36.351584 5413 log.cc:824] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:36.352901 5411 tablet_bootstrap.cc:492] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05: No bootstrap required, opened a new log
I20260926 02:57:36.353163 5411 ts_tablet_manager.cc:1397] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:57:36.354318 5413 tablet_bootstrap.cc:492] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070: No bootstrap required, opened a new log
I20260926 02:57:36.354643 5413 ts_tablet_manager.cc:1397] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 02:57:36.358227 5410 raft_consensus.cc:348] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b703f039cec44db97083aca3b265a05" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45841 } } peers { permanent_uuid: "acc369db30aa4c3bbe99c2051e8d6070" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42669 } } peers { permanent_uuid: "ac190177f1ca42f99e9ed857b3e54470" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45255 } }
I20260926 02:57:36.358783 5410 raft_consensus.cc:374] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:36.358889 5410 raft_consensus.cc:729] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac190177f1ca42f99e9ed857b3e54470, State: Initialized, Role: FOLLOWER
I20260926 02:57:36.359562 5410 consensus_queue.cc:260] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b703f039cec44db97083aca3b265a05" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45841 } } peers { permanent_uuid: "acc369db30aa4c3bbe99c2051e8d6070" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42669 } } peers { permanent_uuid: "ac190177f1ca42f99e9ed857b3e54470" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45255 } }
I20260926 02:57:36.362701 5410 ts_tablet_manager.cc:1428] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260926 02:57:36.363997 5411 raft_consensus.cc:348] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b703f039cec44db97083aca3b265a05" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45841 } } peers { permanent_uuid: "acc369db30aa4c3bbe99c2051e8d6070" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42669 } } peers { permanent_uuid: "ac190177f1ca42f99e9ed857b3e54470" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45255 } }
I20260926 02:57:36.364362 5411 raft_consensus.cc:374] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:36.364482 5411 raft_consensus.cc:729] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b703f039cec44db97083aca3b265a05, State: Initialized, Role: FOLLOWER
I20260926 02:57:36.364292 5413 raft_consensus.cc:348] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b703f039cec44db97083aca3b265a05" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45841 } } peers { permanent_uuid: "acc369db30aa4c3bbe99c2051e8d6070" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42669 } } peers { permanent_uuid: "ac190177f1ca42f99e9ed857b3e54470" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45255 } }
I20260926 02:57:36.364778 5413 raft_consensus.cc:374] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:36.364933 5413 raft_consensus.cc:729] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc369db30aa4c3bbe99c2051e8d6070, State: Initialized, Role: FOLLOWER
I20260926 02:57:36.365197 5411 consensus_queue.cc:260] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b703f039cec44db97083aca3b265a05" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45841 } } peers { permanent_uuid: "acc369db30aa4c3bbe99c2051e8d6070" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42669 } } peers { permanent_uuid: "ac190177f1ca42f99e9ed857b3e54470" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45255 } }
I20260926 02:57:36.365556 5413 consensus_queue.cc:260] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b703f039cec44db97083aca3b265a05" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45841 } } peers { permanent_uuid: "acc369db30aa4c3bbe99c2051e8d6070" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42669 } } peers { permanent_uuid: "ac190177f1ca42f99e9ed857b3e54470" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45255 } }
I20260926 02:57:36.366755 5257 heartbeater.cc:503] Master 127.19.185.254:44853 was elected leader, sending a full tablet report...
I20260926 02:57:36.367549 5413 ts_tablet_manager.cc:1428] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260926 02:57:36.368428 5391 heartbeater.cc:503] Master 127.19.185.254:44853 was elected leader, sending a full tablet report...
I20260926 02:57:36.368856 5411 ts_tablet_manager.cc:1428] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
W20260926 02:57:36.427793 5124 tablet.cc:2369] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:57:36.450543 5417 raft_consensus.cc:482] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:57:36.450881 5417 raft_consensus.cc:504] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b703f039cec44db97083aca3b265a05" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45841 } } peers { permanent_uuid: "acc369db30aa4c3bbe99c2051e8d6070" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42669 } } peers { permanent_uuid: "ac190177f1ca42f99e9ed857b3e54470" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45255 } }
I20260926 02:57:36.452797 5417 leader_election.cc:288] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc369db30aa4c3bbe99c2051e8d6070 (127.19.185.194:42669), ac190177f1ca42f99e9ed857b3e54470 (127.19.185.193:45255)
W20260926 02:57:36.454234 5258 tablet.cc:2369] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:57:36.459610 5394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.453060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:42669 (local address 127.19.185.195:52729)
0926 02:57:36.453387 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:36.453408 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:36.453536 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:36.454020 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:36.454025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:36.454055 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:36.454269 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:36.454282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.455698 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.455704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:36.457343 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:36.457360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.458831 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.458837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:36.459032 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.459066 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:57:36.459165 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 02:57:36.459323 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260926 02:57:36.459729 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.453273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:52729 (local address 127.19.185.194:42669)
0926 02:57:36.453473 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:36.453480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:36.453500 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:57:36.453755 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:36.453763 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:36.453848 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:36.453974 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:36.454431 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.455574 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.457511 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.458686 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.458793 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.458844 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:36.459241 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 02:57:36.459433 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:57:36.460880 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.453420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:55431 (local address 127.19.185.193:45255)
0926 02:57:36.453691 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:36.453698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:36.454124 (+ 426us) server_negotiation.cc:407] Connection header received
0926 02:57:36.454399 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:36.454405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:36.454486 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:36.454572 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:36.455349 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.456437 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.458538 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.459768 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.459916 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.459997 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:36.460489 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 02:57:36.460627 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:57:36.461054 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.453163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:45255 (local address 127.19.185.195:55431)
0926 02:57:36.454025 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:36.454043 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:36.454244 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:36.454733 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:36.454740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:36.454766 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:36.455093 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:36.455104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.456573 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.456576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:36.458382 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:36.458397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.460091 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.460099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:36.460285 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.460329 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 02:57:36.460466 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 02:57:36.460634 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":502,"threads_started":1}
I20260926 02:57:36.460961 5212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfa46ae419954f3684a9dd73a3e8abd6" candidate_uuid: "5b703f039cec44db97083aca3b265a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc369db30aa4c3bbe99c2051e8d6070" is_pre_election: true
I20260926 02:57:36.461576 5212 raft_consensus.cc:2454] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b703f039cec44db97083aca3b265a05 in term 0.
I20260926 02:57:36.462283 5280 leader_election.cc:302] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b703f039cec44db97083aca3b265a05, acc369db30aa4c3bbe99c2051e8d6070; no voters:
I20260926 02:57:36.462261 5078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfa46ae419954f3684a9dd73a3e8abd6" candidate_uuid: "5b703f039cec44db97083aca3b265a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac190177f1ca42f99e9ed857b3e54470" is_pre_election: true
I20260926 02:57:36.462884 5417 raft_consensus.cc:2781] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:57:36.462834 5078 raft_consensus.cc:2454] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b703f039cec44db97083aca3b265a05 in term 0.
I20260926 02:57:36.463037 5417 raft_consensus.cc:482] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:57:36.463142 5417 raft_consensus.cc:3037] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:36.465417 5417 raft_consensus.cc:504] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b703f039cec44db97083aca3b265a05" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45841 } } peers { permanent_uuid: "acc369db30aa4c3bbe99c2051e8d6070" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42669 } } peers { permanent_uuid: "ac190177f1ca42f99e9ed857b3e54470" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45255 } }
I20260926 02:57:36.466010 5417 leader_election.cc:288] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [CANDIDATE]: Term 1 election: Requested vote from peers acc369db30aa4c3bbe99c2051e8d6070 (127.19.185.194:42669), ac190177f1ca42f99e9ed857b3e54470 (127.19.185.193:45255)
I20260926 02:57:36.466470 5078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfa46ae419954f3684a9dd73a3e8abd6" candidate_uuid: "5b703f039cec44db97083aca3b265a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac190177f1ca42f99e9ed857b3e54470"
I20260926 02:57:36.466517 5212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfa46ae419954f3684a9dd73a3e8abd6" candidate_uuid: "5b703f039cec44db97083aca3b265a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc369db30aa4c3bbe99c2051e8d6070"
I20260926 02:57:36.466681 5078 raft_consensus.cc:3037] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:36.466858 5212 raft_consensus.cc:3037] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:36.468953 5078 raft_consensus.cc:2454] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b703f039cec44db97083aca3b265a05 in term 1.
I20260926 02:57:36.469360 5212 raft_consensus.cc:2454] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b703f039cec44db97083aca3b265a05 in term 1.
I20260926 02:57:36.469395 5279 leader_election.cc:302] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b703f039cec44db97083aca3b265a05, ac190177f1ca42f99e9ed857b3e54470; no voters:
I20260926 02:57:36.469727 5417 raft_consensus.cc:2781] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:57:36.470281 5417 raft_consensus.cc:686] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [term 1 LEADER]: Becoming Leader. State: Replica: 5b703f039cec44db97083aca3b265a05, State: Running, Role: LEADER
I20260926 02:57:36.470854 5417 consensus_queue.cc:237] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b703f039cec44db97083aca3b265a05" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45841 } } peers { permanent_uuid: "acc369db30aa4c3bbe99c2051e8d6070" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42669 } } peers { permanent_uuid: "ac190177f1ca42f99e9ed857b3e54470" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45255 } }
W20260926 02:57:36.473909 5392 tablet.cc:2369] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:57:36.475906 4934 catalog_manager.cc:5495] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b703f039cec44db97083aca3b265a05 (127.19.185.195). New cstate: current_term: 1 leader_uuid: "5b703f039cec44db97083aca3b265a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b703f039cec44db97083aca3b265a05" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 45841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc369db30aa4c3bbe99c2051e8d6070" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac190177f1ca42f99e9ed857b3e54470" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45255 } health_report { overall_health: UNKNOWN } } }
I20260926 02:57:36.513372 5405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.507492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.19.185.195:45841)
0926 02:57:36.507697 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:36.507702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:36.507722 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:57:36.507954 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:36.507958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:36.508036 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:36.508164 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:57:36.508614 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.509337 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.510883 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.511373 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.511465 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.512170 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:57:36.512721 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:57:36.512840 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:36.512992 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 02:57:36.513058 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:57:36.521906 5212 raft_consensus.cc:1261] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070 [term 1 FOLLOWER]: Refusing update from remote peer 5b703f039cec44db97083aca3b265a05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:57:36.521903 5078 raft_consensus.cc:1261] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470 [term 1 FOLLOWER]: Refusing update from remote peer 5b703f039cec44db97083aca3b265a05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:57:36.522886 5417 consensus_queue.cc:1035] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac190177f1ca42f99e9ed857b3e54470" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 45255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:57:36.523182 5420 consensus_queue.cc:1035] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc369db30aa4c3bbe99c2051e8d6070" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 42669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:57:36.554512 4934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:57:36.554790 4934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:57:36.558750 4934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b57a5b5c38847b8a6255d31afc2f63d: Sending DeleteTablet for 3 replicas of tablet bfa46ae419954f3684a9dd73a3e8abd6
I20260926 02:57:36.559880 5058 tablet_service.cc:1514] Processing DeleteTablet for tablet bfa46ae419954f3684a9dd73a3e8abd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:36 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:55578
I20260926 02:57:36.559863 5192 tablet_service.cc:1514] Processing DeleteTablet for tablet bfa46ae419954f3684a9dd73a3e8abd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:36 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:42512
I20260926 02:57:36.560557 5326 tablet_service.cc:1514] Processing DeleteTablet for tablet bfa46ae419954f3684a9dd73a3e8abd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:36 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:38650
I20260926 02:57:36.560840 5432 tablet_replica.cc:330] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070: stopping tablet replica
I20260926 02:57:36.562197 5432 raft_consensus.cc:2229] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:57:36.562565 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 4995
I20260926 02:57:36.562824 5432 raft_consensus.cc:2258] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:36.565038 5432 ts_tablet_manager.cc:1905] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:36.567829 5434 tablet_replica.cc:330] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05: stopping tablet replica
I20260926 02:57:36.568307 5434 raft_consensus.cc:2229] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:57:36.568997 5434 raft_consensus.cc:2258] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:36.569806 5433 tablet_replica.cc:330] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470: stopping tablet replica
I20260926 02:57:36.570186 5433 raft_consensus.cc:2229] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:57:36.570680 5433 raft_consensus.cc:2258] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:36.573360 5434 ts_tablet_manager.cc:1905] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:36.573963 5433 ts_tablet_manager.cc:1905] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:36.574509 5432 ts_tablet_manager.cc:1918] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:36.574725 5432 log.cc:1196] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/bfa46ae419954f3684a9dd73a3e8abd6
I20260926 02:57:36.575348 5432 ts_tablet_manager.cc:1939] T bfa46ae419954f3684a9dd73a3e8abd6 P acc369db30aa4c3bbe99c2051e8d6070: Deleting consensus metadata
I20260926 02:57:36.577203 4919 catalog_manager.cc:4841] TS acc369db30aa4c3bbe99c2051e8d6070 (127.19.185.194:42669): tablet bfa46ae419954f3684a9dd73a3e8abd6 (table test-table [id=87686dc8ec8b409dbfdbc7e61460d822]) successfully deleted
I20260926 02:57:36.580583 5433 ts_tablet_manager.cc:1918] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:36.580761 5433 log.cc:1196] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/bfa46ae419954f3684a9dd73a3e8abd6
I20260926 02:57:36.580862 5434 ts_tablet_manager.cc:1918] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:36.581027 5434 log.cc:1196] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/bfa46ae419954f3684a9dd73a3e8abd6
I20260926 02:57:36.581296 5433 ts_tablet_manager.cc:1939] T bfa46ae419954f3684a9dd73a3e8abd6 P ac190177f1ca42f99e9ed857b3e54470: Deleting consensus metadata
I20260926 02:57:36.581570 5434 ts_tablet_manager.cc:1939] T bfa46ae419954f3684a9dd73a3e8abd6 P 5b703f039cec44db97083aca3b265a05: Deleting consensus metadata
I20260926 02:57:36.583112 4918 catalog_manager.cc:4841] TS ac190177f1ca42f99e9ed857b3e54470 (127.19.185.193:45255): tablet bfa46ae419954f3684a9dd73a3e8abd6 (table test-table [id=87686dc8ec8b409dbfdbc7e61460d822]) successfully deleted
I20260926 02:57:36.583261 4919 catalog_manager.cc:4841] TS 5b703f039cec44db97083aca3b265a05 (127.19.185.195:45841): tablet bfa46ae419954f3684a9dd73a3e8abd6 (table test-table [id=87686dc8ec8b409dbfdbc7e61460d822]) successfully deleted
Sep 26 02:57:36 dist-test-slave-ktng krb5kdc[4889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:57:36.597239 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.564104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55580 (local address 127.19.185.193:45255)
0926 02:57:36.564281 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:36.564286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:36.579891 (+ 15605us) server_negotiation.cc:407] Connection header received
0926 02:57:36.580234 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:36.580240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:36.580333 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:36.580437 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:36.582769 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.583748 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.585690 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.586100 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.586195 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.591190 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:36.591203 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:36.591218 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:36.591262 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:36.594230 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:36.594978 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:36.594993 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:36.594996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:36.595132 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:36.595610 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:36.595616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:36.595619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:36.596002 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:36.596323 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:36.596656 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 02:57:36.596894 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:57:36.597908 5118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:36.692107 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 4995
I20260926 02:57:36.710287 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 5129
Sep 26 02:57:36 dist-test-slave-ktng krb5kdc[4889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:57:36.726352 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.711187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.19.185.194:42669)
0926 02:57:36.711396 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:36.711405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:36.711461 (+ 56us) server_negotiation.cc:407] Connection header received
0926 02:57:36.711701 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:36.711705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:36.711779 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:36.711896 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:36.713134 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.713948 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.715506 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.715958 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.716094 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.720358 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:36.720377 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:36.720391 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:36.720428 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:36.723475 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:36.724158 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:36.724173 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:36.724175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:36.724296 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:36.724721 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:36.724729 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:36.724732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:36.725126 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:36.725439 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:36.725738 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 02:57:36.725955 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:57:36.727471 5252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:36.818029 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 5129
I20260926 02:57:36.835539 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 5263
Sep 26 02:57:36 dist-test-slave-ktng krb5kdc[4889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:57:36.854218 5405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:36.836427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.19.185.195:45841)
0926 02:57:36.836585 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:36.836590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:36.836607 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:57:36.836793 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:36.836797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:36.836857 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:36.836948 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:36.838731 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.839668 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.841614 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:36.842058 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:36.842147 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:36.847981 (+ 5834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:36.847995 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:36.848010 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:36.848048 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:36.851050 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:36.851991 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:36.852008 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:36.852010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:36.852152 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:36.852588 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:36.852593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:36.852597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:36.852975 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:36.853295 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:36.853592 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 02:57:36.853823 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 02:57:36.854946 5386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:36.978899 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 5263
I20260926 02:57:36.998932 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 4902
I20260926 02:57:37.000133 4964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:37.112202 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 4902
2026-09-26T02:57:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4410 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T02:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:57:37Z Disabled control of system clock
I20260926 02:57:37.195959 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:34027
--webserver_interface=127.19.185.254
--webserver_port=0
--builtin_ntp_servers=127.19.185.212:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:34027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:57:37.530632 5443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:37.531008 5443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:37.531169 5443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:37.538621 5443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:57:37.538789 5443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:37.538873 5443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:37.538990 5443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:57:37.539070 5443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:57:37.548431 5443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:34027
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:37.549823 5443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:37.551705 5443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:37.563966 5448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:37.564215 5449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:37.564265 5451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:37.565137 5443 server_base.cc:1034] running on GCE node
I20260926 02:57:37.566002 5443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:37.567565 5443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:37.568840 5443 hybrid_clock.cc:648] HybridClock initialized: now 1790391457568792 us; error 53 us; skew 500 ppm
I20260926 02:57:37.569247 5443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:57:37.571929 5443 webserver.cc:460] Webserver started at http://127.19.185.254:37403/ using document root <none> and password file <none>
I20260926 02:57:37.572690 5443 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:37.572835 5443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:37.573182 5443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:37.576333 5443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "e2423ea1d811422ba6ad08d4507c0c59"
format_stamp: "Formatted at 2026-09-26 02:57:37 on dist-test-slave-ktng"
server_key: "ed612b72fcdffc58562500210dae64a0"
server_key_iv: "a2ad9fa0aa20a49c2e647d37e7cf73dd"
server_key_version: "encryptionkey@0"
I20260926 02:57:37.577205 5443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "e2423ea1d811422ba6ad08d4507c0c59"
format_stamp: "Formatted at 2026-09-26 02:57:37 on dist-test-slave-ktng"
server_key: "ed612b72fcdffc58562500210dae64a0"
server_key_iv: "a2ad9fa0aa20a49c2e647d37e7cf73dd"
server_key_version: "encryptionkey@0"
I20260926 02:57:37.582779 5443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:57:37.586942 5457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:37.588461 5443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:57:37.588663 5443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "e2423ea1d811422ba6ad08d4507c0c59"
format_stamp: "Formatted at 2026-09-26 02:57:37 on dist-test-slave-ktng"
server_key: "ed612b72fcdffc58562500210dae64a0"
server_key_iv: "a2ad9fa0aa20a49c2e647d37e7cf73dd"
server_key_version: "encryptionkey@0"
I20260926 02:57:37.588897 5443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:37.601847 5443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:37.602789 5443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:37.603120 5443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:37.623889 5443 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:34027
I20260926 02:57:37.623916 5508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:34027 every 8 connection(s)
I20260926 02:57:37.625324 5443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:57:37.629132 5509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:37.629292 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 5443
I20260926 02:57:37.629480 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:57:37.629958 20199 external_mini_cluster.cc:1428] Setting key c74b0158d6f5d6727c0f2a0b27844e8a
I20260926 02:57:37.642933 5509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59: Bootstrap starting.
I20260926 02:57:37.645035 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:37.631816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.19.185.254:34027)
0926 02:57:37.632831 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:37.632841 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:37.632890 (+ 49us) server_negotiation.cc:407] Connection header received
0926 02:57:37.635254 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:37.635276 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:37.635823 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:37.636157 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:37.636762 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:37.638074 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:37.640148 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:37.640839 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:37.640978 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:37.642217 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:37.642225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:37.642239 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:37.642800 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:37.642929 (+ 129us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:37.642956 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:37.643043 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:37.643406 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 02:57:37.643729 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":358,"threads_started":1}
I20260926 02:57:37.647006 5509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:37.648298 5509 log.cc:824] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:37.651067 5509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59: No bootstrap required, opened a new log
I20260926 02:57:37.658304 5509 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2423ea1d811422ba6ad08d4507c0c59" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34027 } }
I20260926 02:57:37.658681 5509 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:37.658795 5509 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2423ea1d811422ba6ad08d4507c0c59, State: Initialized, Role: FOLLOWER
I20260926 02:57:37.659476 5509 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2423ea1d811422ba6ad08d4507c0c59" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34027 } }
I20260926 02:57:37.659781 5509 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:57:37.659926 5509 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:57:37.660108 5509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:37.662220 5509 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2423ea1d811422ba6ad08d4507c0c59" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34027 } }
I20260926 02:57:37.662678 5509 leader_election.cc:302] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2423ea1d811422ba6ad08d4507c0c59; no voters:
I20260926 02:57:37.663254 5509 leader_election.cc:288] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:57:37.663470 5514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:57:37.664232 5514 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [term 1 LEADER]: Becoming Leader. State: Replica: e2423ea1d811422ba6ad08d4507c0c59, State: Running, Role: LEADER
I20260926 02:57:37.664749 5514 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2423ea1d811422ba6ad08d4507c0c59" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34027 } }
I20260926 02:57:37.665601 5509 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:57:37.667822 5515 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2423ea1d811422ba6ad08d4507c0c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2423ea1d811422ba6ad08d4507c0c59" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34027 } } }
I20260926 02:57:37.668689 5515 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:37.667846 5516 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2423ea1d811422ba6ad08d4507c0c59. Latest consensus state: current_term: 1 leader_uuid: "e2423ea1d811422ba6ad08d4507c0c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2423ea1d811422ba6ad08d4507c0c59" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34027 } } }
I20260926 02:57:37.670856 5516 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59 [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:37.673369 5521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:57:37.678457 5521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:57:37.688705 5521 catalog_manager.cc:1347] Generated new cluster ID: 6188a0b98e584b30b42d6af7626fc2f7
I20260926 02:57:37.688834 5521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:57:37.699236 5521 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:57:37.701227 5521 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:57:37.716045 5521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59: Generated new TSK 0
I20260926 02:57:37.717099 5521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:57:37.733647 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34027
--builtin_ntp_servers=127.19.185.212:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:57:38.089061 5533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:38.089351 5533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:38.089444 5533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:38.097534 5533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:38.097635 5533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:38.097765 5533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:57:38.106904 5533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:38.108196 5533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:38.109915 5533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:38.123757 5538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:38.123699 5541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:38.123795 5539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:38.124655 5533 server_base.cc:1034] running on GCE node
I20260926 02:57:38.125352 5533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:38.126368 5533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:38.127622 5533 hybrid_clock.cc:648] HybridClock initialized: now 1790391458127564 us; error 53 us; skew 500 ppm
I20260926 02:57:38.127969 5533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:57:38.130271 5533 webserver.cc:460] Webserver started at http://127.19.185.193:43317/ using document root <none> and password file <none>
I20260926 02:57:38.130988 5533 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:38.131124 5533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:38.131521 5533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:38.134424 5533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "d7c18a4548744f7dafba05c7844cff59"
format_stamp: "Formatted at 2026-09-26 02:57:38 on dist-test-slave-ktng"
server_key: "499c631330965c36d9b928863a7c78f6"
server_key_iv: "94e3b696e7e6853496347b2147e5c138"
server_key_version: "encryptionkey@0"
I20260926 02:57:38.135224 5533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "d7c18a4548744f7dafba05c7844cff59"
format_stamp: "Formatted at 2026-09-26 02:57:38 on dist-test-slave-ktng"
server_key: "499c631330965c36d9b928863a7c78f6"
server_key_iv: "94e3b696e7e6853496347b2147e5c138"
server_key_version: "encryptionkey@0"
I20260926 02:57:38.140658 5533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:57:38.144359 5547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:38.145815 5533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:57:38.145959 5533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "d7c18a4548744f7dafba05c7844cff59"
format_stamp: "Formatted at 2026-09-26 02:57:38 on dist-test-slave-ktng"
server_key: "499c631330965c36d9b928863a7c78f6"
server_key_iv: "94e3b696e7e6853496347b2147e5c138"
server_key_version: "encryptionkey@0"
I20260926 02:57:38.146139 5533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:38.169509 5533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:38.170339 5533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:38.170612 5533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:38.171808 5533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:38.173460 5533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:38.173557 5533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:38.173632 5533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:38.173696 5533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:38.208112 5533 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:37383
I20260926 02:57:38.208209 5659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:37383 every 8 connection(s)
I20260926 02:57:38.209450 5533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:57:38.213122 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 5533
I20260926 02:57:38.213344 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:57:38.213852 20199 external_mini_cluster.cc:1428] Setting key 63b649391abc761cf39302ac105652dc
I20260926 02:57:38.217082 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34027
--builtin_ntp_servers=127.19.185.212:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:57:38.223701 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:38.212104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:51113 (local address 127.19.185.254:34027)
0926 02:57:38.212609 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:38.212616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:38.213821 (+ 1205us) server_negotiation.cc:407] Connection header received
0926 02:57:38.216147 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:38.216152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:38.216274 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:38.216406 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:38.217484 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:38.218736 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:38.220499 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:38.221209 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:38.221357 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:38.222698 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:38.222705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:38.222712 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:38.222737 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:38.222792 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:38.222811 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:38.222905 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:38.223104 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 02:57:38.223214 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":222,"threads_started":1}
I20260926 02:57:38.224498 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:38.212546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:34027 (local address 127.19.185.193:51113)
0926 02:57:38.213637 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:38.213684 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:38.215848 (+ 2164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:38.216599 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:38.216608 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:38.216699 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:38.217301 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:38.217323 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:38.218927 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:38.218932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:38.220350 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:38.220363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:38.221462 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:38.221468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:38.221670 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:38.222382 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:57:38.222385 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:38.222423 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:57:38.222501 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:38.222908 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:38.222915 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:38.222921 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:57:38.223057 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 02:57:38.223418 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":329,"threads_started":1}
I20260926 02:57:38.226259 5660 heartbeater.cc:347] Connected to a master server at 127.19.185.254:34027
I20260926 02:57:38.226567 5660 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:38.227284 5660 heartbeater.cc:511] Master 127.19.185.254:34027 requested a full tablet report, sending...
I20260926 02:57:38.229461 5474 ts_manager.cc:194] Registered new tserver with Master: d7c18a4548744f7dafba05c7844cff59 (127.19.185.193:37383)
I20260926 02:57:38.230882 5474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.185.193:51113
W20260926 02:57:38.542436 5664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:38.542763 5664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:38.542932 5664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:38.550351 5664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:38.550477 5664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:38.550652 5664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:57:38.559993 5664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:38.561290 5664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:38.563005 5664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:38.576074 5669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:38.576155 5672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:38.576191 5670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:38.577385 5664 server_base.cc:1034] running on GCE node
I20260926 02:57:38.578017 5664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:38.579049 5664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:38.580297 5664 hybrid_clock.cc:648] HybridClock initialized: now 1790391458580244 us; error 33 us; skew 500 ppm
I20260926 02:57:38.580672 5664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:57:38.582921 5664 webserver.cc:460] Webserver started at http://127.19.185.194:37955/ using document root <none> and password file <none>
I20260926 02:57:38.583678 5664 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:38.583770 5664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:38.584031 5664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:38.586944 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "d22ad5091559438ea689bde2e25bad73"
format_stamp: "Formatted at 2026-09-26 02:57:38 on dist-test-slave-ktng"
server_key: "323e97076e331e9b5d789bd20e9b8daa"
server_key_iv: "31c05240440ff296845ced8572b9c25a"
server_key_version: "encryptionkey@0"
I20260926 02:57:38.587791 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "d22ad5091559438ea689bde2e25bad73"
format_stamp: "Formatted at 2026-09-26 02:57:38 on dist-test-slave-ktng"
server_key: "323e97076e331e9b5d789bd20e9b8daa"
server_key_iv: "31c05240440ff296845ced8572b9c25a"
server_key_version: "encryptionkey@0"
I20260926 02:57:38.593485 5664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:57:38.597219 5678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:38.598664 5664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:57:38.598877 5664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "d22ad5091559438ea689bde2e25bad73"
format_stamp: "Formatted at 2026-09-26 02:57:38 on dist-test-slave-ktng"
server_key: "323e97076e331e9b5d789bd20e9b8daa"
server_key_iv: "31c05240440ff296845ced8572b9c25a"
server_key_version: "encryptionkey@0"
I20260926 02:57:38.599063 5664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:38.621389 5664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:38.622190 5664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:38.622450 5664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:38.623587 5664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:38.625104 5664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:38.625187 5664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:38.625321 5664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:38.625454 5664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:38.661760 5664 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:34637
I20260926 02:57:38.661818 5790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:34637 every 8 connection(s)
I20260926 02:57:38.663103 5664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:57:38.663918 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 5664
I20260926 02:57:38.664115 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:57:38.664731 20199 external_mini_cluster.cc:1428] Setting key 1814bd2d441934b17752b1f824b1a780
I20260926 02:57:38.668186 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34027
--builtin_ntp_servers=127.19.185.212:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:57:38.676285 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:38.666005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:39685 (local address 127.19.185.254:34027)
0926 02:57:38.666193 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:38.666200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:38.667978 (+ 1778us) server_negotiation.cc:407] Connection header received
0926 02:57:38.669874 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:38.669880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:38.669965 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:38.670086 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:38.671142 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:38.671979 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:38.673542 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:38.673961 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:38.674094 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:38.675572 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:38.675577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:38.675580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:38.675598 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:38.675632 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:38.675644 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:38.675715 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:38.675991 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 02:57:38.676069 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:57:38.677332 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:38.666772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:34027 (local address 127.19.185.194:39685)
0926 02:57:38.667831 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:38.667874 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:38.669611 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:38.670316 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:38.670325 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:38.670414 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:38.670977 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:38.670993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:38.672126 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:38.672131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:38.673387 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:38.673399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:38.674185 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:38.674193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:38.674430 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:38.675151 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:57:38.675156 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:38.675212 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:57:38.675357 (+ 145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:38.675747 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:38.675761 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:38.675772 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 02:57:38.675973 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 02:57:38.676293 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":375,"threads_started":1}
I20260926 02:57:38.678928 5791 heartbeater.cc:347] Connected to a master server at 127.19.185.254:34027
I20260926 02:57:38.679253 5791 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:38.680262 5791 heartbeater.cc:511] Master 127.19.185.254:34027 requested a full tablet report, sending...
I20260926 02:57:38.681921 5474 ts_manager.cc:194] Registered new tserver with Master: d22ad5091559438ea689bde2e25bad73 (127.19.185.194:34637)
I20260926 02:57:38.682801 5474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.185.194:39685
W20260926 02:57:39.008473 5794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:39.008774 5794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:39.008965 5794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:39.017107 5794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:39.017211 5794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:39.017385 5794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:57:39.026654 5794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:39.028033 5794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:39.029836 5794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:39.043686 5802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:39.043690 5800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:39.043844 5799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:39.045735 5794 server_base.cc:1034] running on GCE node
I20260926 02:57:39.046327 5794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:39.047255 5794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:39.048525 5794 hybrid_clock.cc:648] HybridClock initialized: now 1790391459048484 us; error 42 us; skew 500 ppm
I20260926 02:57:39.048811 5794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:57:39.051220 5794 webserver.cc:460] Webserver started at http://127.19.185.195:32845/ using document root <none> and password file <none>
I20260926 02:57:39.051959 5794 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:39.052050 5794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:39.052361 5794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:39.055392 5794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "26ba9a70e0304554aac389c17532d630"
format_stamp: "Formatted at 2026-09-26 02:57:39 on dist-test-slave-ktng"
server_key: "fb65e2f9bc23b424bd8b2dd24bfa2034"
server_key_iv: "93dc17b836094a800b5c1e8c3438d47e"
server_key_version: "encryptionkey@0"
I20260926 02:57:39.056169 5794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "26ba9a70e0304554aac389c17532d630"
format_stamp: "Formatted at 2026-09-26 02:57:39 on dist-test-slave-ktng"
server_key: "fb65e2f9bc23b424bd8b2dd24bfa2034"
server_key_iv: "93dc17b836094a800b5c1e8c3438d47e"
server_key_version: "encryptionkey@0"
I20260926 02:57:39.061300 5794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:57:39.064769 5808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:39.066360 5794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:57:39.066500 5794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "26ba9a70e0304554aac389c17532d630"
format_stamp: "Formatted at 2026-09-26 02:57:39 on dist-test-slave-ktng"
server_key: "fb65e2f9bc23b424bd8b2dd24bfa2034"
server_key_iv: "93dc17b836094a800b5c1e8c3438d47e"
server_key_version: "encryptionkey@0"
I20260926 02:57:39.066749 5794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:39.093750 5794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:39.094638 5794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:39.094972 5794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:39.096168 5794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:39.097759 5794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:39.097844 5794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:39.097920 5794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:39.097988 5794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:39.133000 5794 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:35505
I20260926 02:57:39.133111 5920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:35505 every 8 connection(s)
I20260926 02:57:39.134420 5794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:57:39.136384 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 5794
I20260926 02:57:39.136574 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:57:39.137118 20199 external_mini_cluster.cc:1428] Setting key d14fc8d396099e0e97a107f861d00a1e
I20260926 02:57:39.148584 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.137731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:41491 (local address 127.19.185.254:34027)
0926 02:57:39.137957 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:39.137964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:39.139834 (+ 1870us) server_negotiation.cc:407] Connection header received
0926 02:57:39.142004 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:39.142009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:39.142098 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:39.142215 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:39.143341 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.144081 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.145642 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.146062 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.146218 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.147621 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:39.147628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:39.147637 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:39.147657 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:39.147690 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:39.147702 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:39.147795 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:39.148116 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 02:57:39.148215 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:57:39.149698 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.138350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:34027 (local address 127.19.185.195:41491)
0926 02:57:39.139657 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:39.139704 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:39.141716 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:39.142462 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:39.142470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:39.142568 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:39.143148 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:39.143165 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.144240 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.144244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:39.145492 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:39.145504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.146308 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.146315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:39.146540 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.147257 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:57:39.147262 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:39.147348 (+ 86us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:57:39.147434 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:39.147903 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:39.147911 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:39.147916 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 02:57:39.148106 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 02:57:39.148503 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":478,"threads_started":1}
I20260926 02:57:39.151358 5921 heartbeater.cc:347] Connected to a master server at 127.19.185.254:34027
I20260926 02:57:39.151742 5921 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:39.152729 5921 heartbeater.cc:511] Master 127.19.185.254:34027 requested a full tablet report, sending...
I20260926 02:57:39.154162 5474 ts_manager.cc:194] Registered new tserver with Master: 26ba9a70e0304554aac389c17532d630 (127.19.185.195:35505)
I20260926 02:57:39.155088 5474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.185.195:41491
I20260926 02:57:39.165463 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:57:39.177529 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.171069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.19.185.254:34027)
0926 02:57:39.171483 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:39.171491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:39.171872 (+ 381us) server_negotiation.cc:407] Connection header received
0926 02:57:39.172085 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:39.172091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:39.172201 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:39.172315 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:39.172856 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.173561 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.175014 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.175542 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.175681 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.176800 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:39.176806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:39.176812 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:39.176835 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:39.176868 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:39.176881 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:39.176975 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:39.177120 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 02:57:39.177214 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260926 02:57:39.181411 5474 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:57:39.188258 5474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:57:39.214717 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.203798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:35505 (local address 127.0.0.1:36884)
0926 02:57:39.206635 (+ 2837us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:39.206657 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:39.206776 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:39.207345 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:39.207349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:39.207375 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:39.207952 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:39.207967 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.209914 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.209919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:39.212156 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:39.212172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.214087 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.214094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:39.214233 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.214255 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 02:57:39.214328 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 02:57:39.214474 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2701,"thread_start_us":2529,"threads_started":1}
I20260926 02:57:39.214969 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.204232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.19.185.195:35505)
0926 02:57:39.206136 (+ 1904us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:39.206145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:39.206768 (+ 623us) server_negotiation.cc:407] Connection header received
0926 02:57:39.206917 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:39.206924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:39.207132 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:39.207352 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:39.208129 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.209731 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.212369 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.213815 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.213983 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.214197 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:39.214410 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 02:57:39.214584 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1753,"thread_start_us":1600,"threads_started":1}
I20260926 02:57:39.215693 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.200970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:34637 (local address 127.0.0.1:44012)
0926 02:57:39.202887 (+ 1917us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:39.202932 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:39.203124 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:39.206092 (+ 2968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:39.206098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:39.206126 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:39.206577 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:39.206589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.208397 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.208403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:39.210698 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:39.210713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.213598 (+ 2885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.213605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:39.213793 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.213849 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 02:57:39.215267 (+ 1418us) client_negotiation.cc:240] Negotiation successful
0926 02:57:39.215420 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1647,"thread_start_us":1518,"threads_started":1}
I20260926 02:57:39.215837 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.201237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44012 (local address 127.19.185.194:34637)
0926 02:57:39.202367 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:39.202378 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:39.203481 (+ 1103us) server_negotiation.cc:407] Connection header received
0926 02:57:39.203600 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:39.203606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:39.203877 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:39.205411 (+ 1534us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:39.206827 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.208227 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.210928 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.212415 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.212567 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.212787 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:39.215341 (+ 2554us) server_negotiation.cc:299] Negotiation successful
0926 02:57:39.215519 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974,"thread_start_us":876,"threads_started":1}
I20260926 02:57:39.216782 5935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.201277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:37383 (local address 127.0.0.1:60128)
0926 02:57:39.203624 (+ 2347us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:39.203644 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:39.203766 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:39.204950 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:39.204961 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:39.205005 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:39.205527 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:39.205543 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.211192 (+ 5649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.211197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:39.213317 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:39.213333 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.216058 (+ 2725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.216064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:39.216265 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.216301 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:57:39.216393 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 02:57:39.216518 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2197,"thread_start_us":2155,"threads_started":1}
I20260926 02:57:39.217190 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.201857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.19.185.193:37383)
0926 02:57:39.203532 (+ 1675us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:39.203547 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:39.203746 (+ 199us) server_negotiation.cc:407] Connection header received
0926 02:57:39.203978 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:39.203990 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:39.204284 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:39.204523 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:39.205768 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.211021 (+ 5253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.214052 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.215543 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.215726 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.215929 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:39.216636 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 02:57:39.216820 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1490,"thread_start_us":1337,"threads_started":1}
I20260926 02:57:39.219148 5856 tablet_service.cc:1467] Processing CreateTablet for tablet 68a5c5d6e6bf4df4921b90546c16bdb5 (DEFAULT_TABLE table=test-table [id=d46da8d6f20347619d167f07c44c1fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:39.219492 5726 tablet_service.cc:1467] Processing CreateTablet for tablet 68a5c5d6e6bf4df4921b90546c16bdb5 (DEFAULT_TABLE table=test-table [id=d46da8d6f20347619d167f07c44c1fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:39.220562 5856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a5c5d6e6bf4df4921b90546c16bdb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:39.220831 5726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a5c5d6e6bf4df4921b90546c16bdb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:39.220691 5595 tablet_service.cc:1467] Processing CreateTablet for tablet 68a5c5d6e6bf4df4921b90546c16bdb5 (DEFAULT_TABLE table=test-table [id=d46da8d6f20347619d167f07c44c1fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:39.221997 5595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a5c5d6e6bf4df4921b90546c16bdb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:39.233790 5660 heartbeater.cc:503] Master 127.19.185.254:34027 was elected leader, sending a full tablet report...
I20260926 02:57:39.239586 5941 tablet_bootstrap.cc:492] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73: Bootstrap starting.
I20260926 02:57:39.240608 5940 tablet_bootstrap.cc:492] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59: Bootstrap starting.
I20260926 02:57:39.243544 5941 tablet_bootstrap.cc:654] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:39.243901 5939 tablet_bootstrap.cc:492] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630: Bootstrap starting.
I20260926 02:57:39.244349 5940 tablet_bootstrap.cc:654] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:39.244889 5941 log.cc:824] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:39.245615 5940 log.cc:824] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:39.246582 5939 tablet_bootstrap.cc:654] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:39.247686 5939 log.cc:824] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:39.251637 5940 tablet_bootstrap.cc:492] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59: No bootstrap required, opened a new log
I20260926 02:57:39.251843 5939 tablet_bootstrap.cc:492] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630: No bootstrap required, opened a new log
I20260926 02:57:39.251952 5940 ts_tablet_manager.cc:1397] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260926 02:57:39.252027 5941 tablet_bootstrap.cc:492] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73: No bootstrap required, opened a new log
I20260926 02:57:39.252103 5939 ts_tablet_manager.cc:1397] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260926 02:57:39.252341 5941 ts_tablet_manager.cc:1397] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20260926 02:57:39.259670 5940 raft_consensus.cc:348] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22ad5091559438ea689bde2e25bad73" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34637 } } peers { permanent_uuid: "d7c18a4548744f7dafba05c7844cff59" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37383 } } peers { permanent_uuid: "26ba9a70e0304554aac389c17532d630" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35505 } }
I20260926 02:57:39.260067 5940 raft_consensus.cc:374] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:39.260214 5940 raft_consensus.cc:729] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c18a4548744f7dafba05c7844cff59, State: Initialized, Role: FOLLOWER
I20260926 02:57:39.260855 5940 consensus_queue.cc:260] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22ad5091559438ea689bde2e25bad73" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34637 } } peers { permanent_uuid: "d7c18a4548744f7dafba05c7844cff59" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37383 } } peers { permanent_uuid: "26ba9a70e0304554aac389c17532d630" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35505 } }
I20260926 02:57:39.262537 5940 ts_tablet_manager.cc:1428] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260926 02:57:39.262867 5941 raft_consensus.cc:348] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22ad5091559438ea689bde2e25bad73" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34637 } } peers { permanent_uuid: "d7c18a4548744f7dafba05c7844cff59" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37383 } } peers { permanent_uuid: "26ba9a70e0304554aac389c17532d630" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35505 } }
I20260926 02:57:39.263391 5941 raft_consensus.cc:374] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:39.263509 5941 raft_consensus.cc:729] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22ad5091559438ea689bde2e25bad73, State: Initialized, Role: FOLLOWER
I20260926 02:57:39.264060 5941 consensus_queue.cc:260] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22ad5091559438ea689bde2e25bad73" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34637 } } peers { permanent_uuid: "d7c18a4548744f7dafba05c7844cff59" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37383 } } peers { permanent_uuid: "26ba9a70e0304554aac389c17532d630" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35505 } }
I20260926 02:57:39.263944 5939 raft_consensus.cc:348] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22ad5091559438ea689bde2e25bad73" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34637 } } peers { permanent_uuid: "d7c18a4548744f7dafba05c7844cff59" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37383 } } peers { permanent_uuid: "26ba9a70e0304554aac389c17532d630" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35505 } }
I20260926 02:57:39.264355 5939 raft_consensus.cc:374] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:39.264462 5939 raft_consensus.cc:729] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ba9a70e0304554aac389c17532d630, State: Initialized, Role: FOLLOWER
I20260926 02:57:39.265192 5791 heartbeater.cc:503] Master 127.19.185.254:34027 was elected leader, sending a full tablet report...
I20260926 02:57:39.265071 5939 consensus_queue.cc:260] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22ad5091559438ea689bde2e25bad73" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34637 } } peers { permanent_uuid: "d7c18a4548744f7dafba05c7844cff59" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37383 } } peers { permanent_uuid: "26ba9a70e0304554aac389c17532d630" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35505 } }
I20260926 02:57:39.265882 5941 ts_tablet_manager.cc:1428] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260926 02:57:39.266820 5921 heartbeater.cc:503] Master 127.19.185.254:34027 was elected leader, sending a full tablet report...
I20260926 02:57:39.267423 5939 ts_tablet_manager.cc:1428] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
W20260926 02:57:39.386339 5922 tablet.cc:2369] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:57:39.415781 5792 tablet.cc:2369] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:57:39.461895 5661 tablet.cc:2369] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:57:39.487869 5945 raft_consensus.cc:482] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:57:39.488256 5945 raft_consensus.cc:504] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22ad5091559438ea689bde2e25bad73" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34637 } } peers { permanent_uuid: "d7c18a4548744f7dafba05c7844cff59" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37383 } } peers { permanent_uuid: "26ba9a70e0304554aac389c17532d630" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35505 } }
I20260926 02:57:39.490061 5945 leader_election.cc:288] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d22ad5091559438ea689bde2e25bad73 (127.19.185.194:34637), 26ba9a70e0304554aac389c17532d630 (127.19.185.195:35505)
I20260926 02:57:39.498946 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.490708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:44297 (local address 127.19.185.195:35505)
0926 02:57:39.491146 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:39.491154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:39.491539 (+ 385us) server_negotiation.cc:407] Connection header received
0926 02:57:39.491745 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:39.491751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:39.491840 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:39.491941 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:39.492780 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.493684 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.496936 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.497909 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.498073 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.498139 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:39.498565 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 02:57:39.498698 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
I20260926 02:57:39.499079 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.490410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:35505 (local address 127.19.185.193:44297)
0926 02:57:39.491385 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:39.491409 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:39.491598 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:39.492018 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:39.492025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:39.492060 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:39.492594 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:39.492611 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.493914 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.493920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:39.496669 (+ 2749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:39.496683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.498119 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.498136 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:39.498332 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.498409 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 02:57:39.498518 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 02:57:39.498680 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":199,"thread_start_us":473,"threads_started":1}
I20260926 02:57:39.499066 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.490610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:36619 (local address 127.19.185.194:34637)
0926 02:57:39.490979 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:39.490988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:39.491973 (+ 985us) server_negotiation.cc:407] Connection header received
0926 02:57:39.492083 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:39.492091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:39.492198 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:39.492365 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:39.493093 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.494129 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.496873 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.497822 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.498009 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.498094 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:39.498663 (+ 569us) server_negotiation.cc:299] Negotiation successful
0926 02:57:39.498805 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260926 02:57:39.499989 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.490551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:34637 (local address 127.19.185.193:36619)
0926 02:57:39.491805 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:39.491829 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:39.491966 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:39.492383 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:39.492386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:39.492411 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:39.492899 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:39.492915 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.494265 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.494269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:39.496664 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:39.496678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.498215 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.498222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:39.498451 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.498500 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 02:57:39.499408 (+ 908us) client_negotiation.cc:240] Negotiation successful
0926 02:57:39.499594 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"mutex_wait_us":125,"thread_start_us":809,"threads_started":1}
I20260926 02:57:39.500416 5876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a5c5d6e6bf4df4921b90546c16bdb5" candidate_uuid: "d7c18a4548744f7dafba05c7844cff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26ba9a70e0304554aac389c17532d630" is_pre_election: true
I20260926 02:57:39.500919 5876 raft_consensus.cc:2454] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c18a4548744f7dafba05c7844cff59 in term 0.
I20260926 02:57:39.501385 5746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a5c5d6e6bf4df4921b90546c16bdb5" candidate_uuid: "d7c18a4548744f7dafba05c7844cff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22ad5091559438ea689bde2e25bad73" is_pre_election: true
I20260926 02:57:39.501603 5549 leader_election.cc:302] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ba9a70e0304554aac389c17532d630, d7c18a4548744f7dafba05c7844cff59; no voters:
I20260926 02:57:39.502002 5746 raft_consensus.cc:2454] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c18a4548744f7dafba05c7844cff59 in term 0.
I20260926 02:57:39.502069 5945 raft_consensus.cc:2781] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:57:39.502219 5945 raft_consensus.cc:482] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:57:39.502349 5945 raft_consensus.cc:3037] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:39.505046 5945 raft_consensus.cc:504] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22ad5091559438ea689bde2e25bad73" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34637 } } peers { permanent_uuid: "d7c18a4548744f7dafba05c7844cff59" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37383 } } peers { permanent_uuid: "26ba9a70e0304554aac389c17532d630" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35505 } }
I20260926 02:57:39.505728 5945 leader_election.cc:288] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [CANDIDATE]: Term 1 election: Requested vote from peers d22ad5091559438ea689bde2e25bad73 (127.19.185.194:34637), 26ba9a70e0304554aac389c17532d630 (127.19.185.195:35505)
I20260926 02:57:39.506090 5746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a5c5d6e6bf4df4921b90546c16bdb5" candidate_uuid: "d7c18a4548744f7dafba05c7844cff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22ad5091559438ea689bde2e25bad73"
I20260926 02:57:39.506225 5876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a5c5d6e6bf4df4921b90546c16bdb5" candidate_uuid: "d7c18a4548744f7dafba05c7844cff59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26ba9a70e0304554aac389c17532d630"
I20260926 02:57:39.506304 5746 raft_consensus.cc:3037] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:39.506471 5876 raft_consensus.cc:3037] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:39.508800 5876 raft_consensus.cc:2454] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c18a4548744f7dafba05c7844cff59 in term 1.
I20260926 02:57:39.508862 5746 raft_consensus.cc:2454] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c18a4548744f7dafba05c7844cff59 in term 1.
I20260926 02:57:39.509228 5549 leader_election.cc:302] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ba9a70e0304554aac389c17532d630, d7c18a4548744f7dafba05c7844cff59; no voters:
I20260926 02:57:39.509521 5945 raft_consensus.cc:2781] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:57:39.510032 5945 raft_consensus.cc:686] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [term 1 LEADER]: Becoming Leader. State: Replica: d7c18a4548744f7dafba05c7844cff59, State: Running, Role: LEADER
I20260926 02:57:39.510655 5945 consensus_queue.cc:237] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22ad5091559438ea689bde2e25bad73" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34637 } } peers { permanent_uuid: "d7c18a4548744f7dafba05c7844cff59" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37383 } } peers { permanent_uuid: "26ba9a70e0304554aac389c17532d630" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35505 } }
I20260926 02:57:39.515491 5474 catalog_manager.cc:5495] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7c18a4548744f7dafba05c7844cff59 (127.19.185.193). New cstate: current_term: 1 leader_uuid: "d7c18a4548744f7dafba05c7844cff59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22ad5091559438ea689bde2e25bad73" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c18a4548744f7dafba05c7844cff59" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26ba9a70e0304554aac389c17532d630" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35505 } health_report { overall_health: UNKNOWN } } }
I20260926 02:57:39.582404 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.577016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.19.185.193:37383)
0926 02:57:39.577246 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:39.577251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:39.577269 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:57:39.577362 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:39.577366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:39.577454 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:39.577632 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:57:39.577943 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.578687 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.580301 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.580712 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.580836 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.581300 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:57:39.581823 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:57:39.581946 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:39.582029 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 02:57:39.582093 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:57:39.591419 5876 raft_consensus.cc:1261] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630 [term 1 FOLLOWER]: Refusing update from remote peer d7c18a4548744f7dafba05c7844cff59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:57:39.591419 5746 raft_consensus.cc:1261] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73 [term 1 FOLLOWER]: Refusing update from remote peer d7c18a4548744f7dafba05c7844cff59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:57:39.592438 5950 consensus_queue.cc:1035] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d22ad5091559438ea689bde2e25bad73" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 34637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:57:39.592883 5945 consensus_queue.cc:1035] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26ba9a70e0304554aac389c17532d630" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 35505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:57:39.608407 5954 mvcc.cc:205] Tried to move back new op lower bound from 7333443418474835968 to 7333443418160185344. Current Snapshot: MvccSnapshot[applied={T|T < 7333443418474835968 or (T in {7333443418474835968})}]
I20260926 02:57:39.621447 5473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:57:39.621770 5473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:57:39.625641 5473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2423ea1d811422ba6ad08d4507c0c59: Sending DeleteTablet for 3 replicas of tablet 68a5c5d6e6bf4df4921b90546c16bdb5
I20260926 02:57:39.626840 5726 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a5c5d6e6bf4df4921b90546c16bdb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:39 UTC) from {username='slave'} at 127.0.0.1:44012
I20260926 02:57:39.626855 5595 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a5c5d6e6bf4df4921b90546c16bdb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:39 UTC) from {username='slave'} at 127.0.0.1:60128
I20260926 02:57:39.629792 5856 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a5c5d6e6bf4df4921b90546c16bdb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:39 UTC) from {username='slave'} at 127.0.0.1:36884
I20260926 02:57:39.631640 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 5533
I20260926 02:57:39.640533 5965 tablet_replica.cc:330] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630: stopping tablet replica
I20260926 02:57:39.641012 5963 tablet_replica.cc:330] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59: stopping tablet replica
I20260926 02:57:39.641059 5965 raft_consensus.cc:2229] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:57:39.641443 5963 raft_consensus.cc:2229] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:57:39.641624 5965 raft_consensus.cc:2258] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:39.642976 5964 tablet_replica.cc:330] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73: stopping tablet replica
I20260926 02:57:39.643394 5964 raft_consensus.cc:2229] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:57:39.643916 5964 raft_consensus.cc:2258] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:39.642262 5963 raft_consensus.cc:2258] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:39.646565 5964 ts_tablet_manager.cc:1905] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:39.648036 5963 ts_tablet_manager.cc:1905] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:39.648324 5965 ts_tablet_manager.cc:1905] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:39.658517 5965 ts_tablet_manager.cc:1918] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:39.658722 5965 log.cc:1196] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/68a5c5d6e6bf4df4921b90546c16bdb5
I20260926 02:57:39.659183 5965 ts_tablet_manager.cc:1939] T 68a5c5d6e6bf4df4921b90546c16bdb5 P 26ba9a70e0304554aac389c17532d630: Deleting consensus metadata
I20260926 02:57:39.660902 5459 catalog_manager.cc:4841] TS 26ba9a70e0304554aac389c17532d630 (127.19.185.195:35505): tablet 68a5c5d6e6bf4df4921b90546c16bdb5 (table test-table [id=d46da8d6f20347619d167f07c44c1fe3]) successfully deleted
I20260926 02:57:39.665405 5963 ts_tablet_manager.cc:1918] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:39.665515 5964 ts_tablet_manager.cc:1918] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:39.665692 5963 log.cc:1196] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/68a5c5d6e6bf4df4921b90546c16bdb5
I20260926 02:57:39.665832 5964 log.cc:1196] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/68a5c5d6e6bf4df4921b90546c16bdb5
I20260926 02:57:39.666489 5964 ts_tablet_manager.cc:1939] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d22ad5091559438ea689bde2e25bad73: Deleting consensus metadata
I20260926 02:57:39.666709 5963 ts_tablet_manager.cc:1939] T 68a5c5d6e6bf4df4921b90546c16bdb5 P d7c18a4548744f7dafba05c7844cff59: Deleting consensus metadata
I20260926 02:57:39.669279 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.632867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.19.185.193:37383)
0926 02:57:39.633028 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:39.633035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:39.646462 (+ 13427us) server_negotiation.cc:407] Connection header received
0926 02:57:39.648598 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:39.648604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:39.648716 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:39.648788 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:39.649507 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.650785 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.660758 (+ 9973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.661413 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.661500 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.667523 (+ 6023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:39.667536 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:39.667555 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:39.668341 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:39.668407 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:39.668434 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:39.668517 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:39.668767 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 02:57:39.668904 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 02:57:39.670065 5461 catalog_manager.cc:4841] TS d22ad5091559438ea689bde2e25bad73 (127.19.185.194:34637): tablet 68a5c5d6e6bf4df4921b90546c16bdb5 (table test-table [id=d46da8d6f20347619d167f07c44c1fe3]) successfully deleted
I20260926 02:57:39.670614 5655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:39.671129 5459 catalog_manager.cc:4841] TS d7c18a4548744f7dafba05c7844cff59 (127.19.185.193:37383): tablet 68a5c5d6e6bf4df4921b90546c16bdb5 (table test-table [id=d46da8d6f20347619d167f07c44c1fe3]) successfully deleted
I20260926 02:57:39.783736 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 5533
I20260926 02:57:39.802474 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 5664
I20260926 02:57:39.809746 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.803243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.19.185.194:34637)
0926 02:57:39.803475 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:39.803483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:39.803553 (+ 70us) server_negotiation.cc:407] Connection header received
0926 02:57:39.803767 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:39.803772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:39.803869 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:39.804003 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:39.804404 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.805149 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.806440 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.806873 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.807016 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.808266 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:39.808276 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:39.808291 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:39.808928 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:39.808980 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:39.809001 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:39.809132 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:39.809266 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 02:57:39.809378 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:57:39.810412 5786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:39.899096 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 5664
I20260926 02:57:39.918147 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 5794
I20260926 02:57:39.926023 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:39.919034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36886 (local address 127.19.185.195:35505)
0926 02:57:39.919249 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:39.919255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:39.919277 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:57:39.919505 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:39.919516 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:39.919611 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:39.919743 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:39.920234 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.921028 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.922521 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:39.922970 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:39.923140 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:39.924525 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:39.924536 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:39.924552 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:39.925176 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:39.925239 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:39.925260 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:39.925400 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:39.925546 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 02:57:39.925662 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:57:39.926714 5916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:40.016587 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 5794
I20260926 02:57:40.033457 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 5443
I20260926 02:57:40.034651 5504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:40.145941 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 5443
2026-09-26T02:57:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3030 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:57:40 dist-test-slave-ktng krb5kdc[5973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:57:40 dist-test-slave-ktng krb5kdc[5973](info): set up 2 sockets
Sep 26 02:57:40 dist-test-slave-ktng krb5kdc[5973](info): commencing operation
krb5kdc: starting...
W20260926 02:57:41.272539 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:57:41 dist-test-slave-ktng krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:57:41.534868 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:39065
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:39065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:41.874368 5986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:41.874673 5986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:41.874773 5986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:41.882266 5986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:57:41.882380 5986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:41.882427 5986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:41.882478 5986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:57:41.882524 5986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:57:41.892709 5986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:39065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:39065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:41.894038 5986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:41.895329 5986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:41.907689 5991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:41.907837 5992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:41.907845 5994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:41.909044 5986 server_base.cc:1034] running on GCE node
I20260926 02:57:41.909900 5986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:41.911442 5986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:41.912787 5986 hybrid_clock.cc:648] HybridClock initialized: now 1790391461912668 us; error 57 us; skew 500 ppm
I20260926 02:57:41.913178 5986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:41 dist-test-slave-ktng krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391461, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:41.916472 5986 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:57:41.918169 5986 webserver.cc:460] Webserver started at http://127.19.185.254:41499/ using document root <none> and password file <none>
I20260926 02:57:41.918968 5986 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:41.919137 5986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:41.919504 5986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:41.922444 5986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "11ad29df20de41ffb83cc030ce22a54a"
format_stamp: "Formatted at 2026-09-26 02:57:41 on dist-test-slave-ktng"
server_key: "d1763c83feff7a24517249582e7ff513"
server_key_iv: "9464a93b57062f3941e03be1cd943f99"
server_key_version: "encryptionkey@0"
I20260926 02:57:41.923214 5986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "11ad29df20de41ffb83cc030ce22a54a"
format_stamp: "Formatted at 2026-09-26 02:57:41 on dist-test-slave-ktng"
server_key: "d1763c83feff7a24517249582e7ff513"
server_key_iv: "9464a93b57062f3941e03be1cd943f99"
server_key_version: "encryptionkey@0"
I20260926 02:57:41.928648 5986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:57:41.932403 6001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:41.933866 5986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:57:41.934068 5986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "11ad29df20de41ffb83cc030ce22a54a"
format_stamp: "Formatted at 2026-09-26 02:57:41 on dist-test-slave-ktng"
server_key: "d1763c83feff7a24517249582e7ff513"
server_key_iv: "9464a93b57062f3941e03be1cd943f99"
server_key_version: "encryptionkey@0"
I20260926 02:57:41.934269 5986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:41.963008 5986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:41.967635 5986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:41.967949 5986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:41.987540 5986 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:39065
I20260926 02:57:41.987547 6052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:39065 every 8 connection(s)
I20260926 02:57:41.988987 5986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:57:41.989625 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 5986
I20260926 02:57:41.989845 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:57:41.990475 20199 external_mini_cluster.cc:1428] Setting key fb5c16a9d4d5500e7b5863720455df39
I20260926 02:57:41.993716 6053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:57:42 dist-test-slave-ktng krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:57:42.007009 6053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a: Bootstrap starting.
I20260926 02:57:42.011425 6053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:42.012753 6053 log.cc:824] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:42.016455 6053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a: No bootstrap required, opened a new log
I20260926 02:57:42.017375 6056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:41.992539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.19.185.254:39065)
0926 02:57:41.993362 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:41.993379 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:41.993428 (+ 49us) server_negotiation.cc:407] Connection header received
0926 02:57:41.995836 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:41.995867 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:41.996376 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:41.996871 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:41.998490 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:42.000665 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:42.002110 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:42.003188 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:42.003501 (+ 313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:42.007975 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:42.007985 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:42.008004 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:42.008053 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:42.012405 (+ 4352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:42.013248 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:42.013261 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:42.013274 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:42.013534 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:42.014006 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:42.014010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:42.014012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:42.014734 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:42.015010 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:42.015485 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 02:57:42.015933 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":324,"threads_started":1}
I20260926 02:57:42.024863 6053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ad29df20de41ffb83cc030ce22a54a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39065 } }
I20260926 02:57:42.025202 6053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:42.025269 6053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ad29df20de41ffb83cc030ce22a54a, State: Initialized, Role: FOLLOWER
I20260926 02:57:42.025808 6053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ad29df20de41ffb83cc030ce22a54a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39065 } }
I20260926 02:57:42.026034 6053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:57:42.026165 6053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:57:42.026352 6053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:42.028704 6053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ad29df20de41ffb83cc030ce22a54a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39065 } }
I20260926 02:57:42.029222 6053 leader_election.cc:302] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11ad29df20de41ffb83cc030ce22a54a; no voters:
I20260926 02:57:42.029805 6053 leader_election.cc:288] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:57:42.029969 6058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:57:42.030777 6058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [term 1 LEADER]: Becoming Leader. State: Replica: 11ad29df20de41ffb83cc030ce22a54a, State: Running, Role: LEADER
I20260926 02:57:42.031476 6058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ad29df20de41ffb83cc030ce22a54a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39065 } }
I20260926 02:57:42.032339 6053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:57:42.034626 6059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11ad29df20de41ffb83cc030ce22a54a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ad29df20de41ffb83cc030ce22a54a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39065 } } }
I20260926 02:57:42.035177 6059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:42.039559 6060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11ad29df20de41ffb83cc030ce22a54a. Latest consensus state: current_term: 1 leader_uuid: "11ad29df20de41ffb83cc030ce22a54a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ad29df20de41ffb83cc030ce22a54a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39065 } } }
I20260926 02:57:42.039960 6060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a [sys.catalog]: This master's current role is: LEADER
W20260926 02:57:42.043828 6074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:57:42.043931 6074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:57:42.044073 6067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:57:42.046391 6067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:57:42.055509 6067 catalog_manager.cc:1347] Generated new cluster ID: 74a810aa7ea84f838dd8341457c4394b
I20260926 02:57:42.055640 6067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:57:42.074924 6067 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:57:42.076357 6067 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:57:42.093964 6067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a: Generated new TSK 0
I20260926 02:57:42.094869 6067 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:57:42.174800 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39065
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:42.519939 6079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:42.520206 6079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:42.520346 6079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:42.527763 6079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:42.527870 6079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:42.528069 6079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:57:42.538102 6079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:42.539636 6079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:42.541010 6079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:42.554009 6084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:42.554137 6085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:42.554301 6087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:42.555372 6079 server_base.cc:1034] running on GCE node
I20260926 02:57:42.555997 6079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:42.556882 6079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:42.558091 6079 hybrid_clock.cc:648] HybridClock initialized: now 1790391462558054 us; error 33 us; skew 500 ppm
I20260926 02:57:42.558386 6079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:42 dist-test-slave-ktng krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391462, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:42.561566 6079 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:57:42.563078 6079 webserver.cc:460] Webserver started at http://127.19.185.193:35927/ using document root <none> and password file <none>
I20260926 02:57:42.563792 6079 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:42.563896 6079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:42.564230 6079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:42.567085 6079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "258acff31d2e4794a59d690303211722"
format_stamp: "Formatted at 2026-09-26 02:57:42 on dist-test-slave-ktng"
server_key: "e252159e3d0b1e86f77ace7b3fdf1369"
server_key_iv: "8c286da7a22baee27fccac69a5806014"
server_key_version: "encryptionkey@0"
I20260926 02:57:42.567858 6079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "258acff31d2e4794a59d690303211722"
format_stamp: "Formatted at 2026-09-26 02:57:42 on dist-test-slave-ktng"
server_key: "e252159e3d0b1e86f77ace7b3fdf1369"
server_key_iv: "8c286da7a22baee27fccac69a5806014"
server_key_version: "encryptionkey@0"
I20260926 02:57:42.573002 6079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:57:42.576560 6094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:42.577962 6079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:57:42.578102 6079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "258acff31d2e4794a59d690303211722"
format_stamp: "Formatted at 2026-09-26 02:57:42 on dist-test-slave-ktng"
server_key: "e252159e3d0b1e86f77ace7b3fdf1369"
server_key_iv: "8c286da7a22baee27fccac69a5806014"
server_key_version: "encryptionkey@0"
I20260926 02:57:42.578261 6079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:42.615449 6079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:42.619778 6079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:42.620038 6079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:42.621004 6079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:42.622349 6079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:42.622429 6079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:42.622514 6079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:42.622558 6079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:42.657771 6079 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:33949
I20260926 02:57:42.657869 6206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:33949 every 8 connection(s)
I20260926 02:57:42.659123 6079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:57:42.666224 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 6079
I20260926 02:57:42.666442 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:57:42.667065 20199 external_mini_cluster.cc:1428] Setting key c8783fb4172134acdd50e45115f53943
Sep 26 02:57:42 dist-test-slave-ktng krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391462, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:57:42.683465 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:42.661947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:55655 (local address 127.19.185.254:39065)
0926 02:57:42.662394 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:42.662398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:42.663528 (+ 1130us) server_negotiation.cc:407] Connection header received
0926 02:57:42.666333 (+ 2805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:42.666340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:42.666430 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:42.666532 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:42.668626 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:42.669522 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:42.672104 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:42.672517 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:42.672613 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:42.677028 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:42.677034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:42.677037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:42.677073 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:42.679890 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:42.680706 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:42.680710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:42.680712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:42.680834 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:42.681322 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:42.681330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:42.681333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:42.681711 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:42.681895 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:42.682784 (+ 889us) server_negotiation.cc:299] Negotiation successful
0926 02:57:42.683047 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":257,"threads_started":1}
I20260926 02:57:42.684608 6210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:42.662374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39065 (local address 127.19.185.193:55655)
0926 02:57:42.663356 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:42.663409 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:42.665849 (+ 2440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:42.666981 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:42.666994 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:42.667711 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:42.668446 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:42.668471 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:42.669684 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:42.669692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:42.671941 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:42.671959 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:42.672764 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:42.672774 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:42.673105 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:42.673873 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:57:42.673908 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:42.676602 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:42.680041 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:42.680051 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:42.680069 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:42.680554 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:42.680948 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:42.680953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:42.680957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:42.681191 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:42.681879 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:42.681887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:42.682334 (+ 447us) client_negotiation.cc:769] Sending connection context
0926 02:57:42.682668 (+ 334us) client_negotiation.cc:240] Negotiation successful
0926 02:57:42.683260 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":492,"threads_started":1}
I20260926 02:57:42.686553 6207 heartbeater.cc:347] Connected to a master server at 127.19.185.254:39065
I20260926 02:57:42.686872 6207 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:42.687736 6207 heartbeater.cc:511] Master 127.19.185.254:39065 requested a full tablet report, sending...
I20260926 02:57:42.690701 6018 ts_manager.cc:194] Registered new tserver with Master: 258acff31d2e4794a59d690303211722 (127.19.185.193:33949)
I20260926 02:57:42.693050 6018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:55655
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:57:42.731376 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39065
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:43.070413 6213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:43.070763 6213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:43.070945 6213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:43.078435 6213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:43.078599 6213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:43.078783 6213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:57:43.089205 6213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:43.090556 6213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:43.091998 6213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:43.107795 6218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:43.107911 6221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:43.107816 6219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:43.109234 6213 server_base.cc:1034] running on GCE node
I20260926 02:57:43.109938 6213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:43.111092 6213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:43.112341 6213 hybrid_clock.cc:648] HybridClock initialized: now 1790391463112284 us; error 41 us; skew 500 ppm
I20260926 02:57:43.112684 6213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:43 dist-test-slave-ktng krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391463, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:43.116091 6213 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:57:43.117738 6213 webserver.cc:460] Webserver started at http://127.19.185.194:32775/ using document root <none> and password file <none>
I20260926 02:57:43.118445 6213 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:43.118595 6213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:43.118916 6213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:43.121923 6213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "250719e0e2394e2b877cf794ec2d2781"
format_stamp: "Formatted at 2026-09-26 02:57:43 on dist-test-slave-ktng"
server_key: "aa542582541fa0d0834bf9a92fcc5e47"
server_key_iv: "a6bbbce2e7c813384bbb9137580b2a47"
server_key_version: "encryptionkey@0"
I20260926 02:57:43.122725 6213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "250719e0e2394e2b877cf794ec2d2781"
format_stamp: "Formatted at 2026-09-26 02:57:43 on dist-test-slave-ktng"
server_key: "aa542582541fa0d0834bf9a92fcc5e47"
server_key_iv: "a6bbbce2e7c813384bbb9137580b2a47"
server_key_version: "encryptionkey@0"
I20260926 02:57:43.128005 6213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:57:43.131592 6228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:43.133224 6213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:57:43.133419 6213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "250719e0e2394e2b877cf794ec2d2781"
format_stamp: "Formatted at 2026-09-26 02:57:43 on dist-test-slave-ktng"
server_key: "aa542582541fa0d0834bf9a92fcc5e47"
server_key_iv: "a6bbbce2e7c813384bbb9137580b2a47"
server_key_version: "encryptionkey@0"
I20260926 02:57:43.133665 6213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:43.152968 6213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:43.157253 6213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:43.157585 6213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:43.158603 6213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:43.160040 6213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:43.160156 6213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:43.160326 6213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:43.160401 6213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:43.198664 6213 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:33339
I20260926 02:57:43.198712 6340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:33339 every 8 connection(s)
I20260926 02:57:43.200009 6213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:57:43.207878 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 6213
I20260926 02:57:43.208097 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:57:43.208544 20199 external_mini_cluster.cc:1428] Setting key 807e0fa87e358afaa961d38305e6746d
Sep 26 02:57:43 dist-test-slave-ktng krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391463, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:57:43.222249 6343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.202717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:51099 (local address 127.19.185.254:39065)
0926 02:57:43.203341 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:43.203351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:43.204247 (+ 896us) server_negotiation.cc:407] Connection header received
0926 02:57:43.206138 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:43.206143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:43.206237 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:43.206360 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:43.208280 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.209084 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.210846 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.211520 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.211677 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.215088 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:43.215097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:43.215104 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:43.215152 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:43.218879 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:43.219677 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:43.219683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:43.219687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:43.219840 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:43.220367 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:43.220370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:43.220372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:43.220633 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:43.220748 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:43.221790 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0926 02:57:43.221970 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":343,"threads_started":1}
I20260926 02:57:43.224068 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.203376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39065 (local address 127.19.185.194:51099)
0926 02:57:43.204082 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:43.204120 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:43.205858 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:43.206604 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:43.206613 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:43.207145 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:43.208069 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:43.208106 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.209257 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.209265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:43.210703 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:43.210715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.211731 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.211736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:43.211994 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.212700 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:57:43.212726 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:43.214862 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:43.219025 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:43.219032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:43.219045 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:43.219560 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:43.219956 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:43.219963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:43.219968 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:43.220229 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:43.220823 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:43.220834 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:43.221308 (+ 474us) client_negotiation.cc:769] Sending connection context
0926 02:57:43.221615 (+ 307us) client_negotiation.cc:240] Negotiation successful
0926 02:57:43.222198 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":270,"threads_started":1}
I20260926 02:57:43.225651 6341 heartbeater.cc:347] Connected to a master server at 127.19.185.254:39065
I20260926 02:57:43.226034 6341 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:43.226795 6341 heartbeater.cc:511] Master 127.19.185.254:39065 requested a full tablet report, sending...
I20260926 02:57:43.228365 6018 ts_manager.cc:194] Registered new tserver with Master: 250719e0e2394e2b877cf794ec2d2781 (127.19.185.194:33339)
I20260926 02:57:43.229166 6018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:51099
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:57:43.266986 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39065
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:43.594944 6347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:43.595311 6347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:43.595484 6347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:43.602716 6347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:43.602890 6347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:43.603087 6347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:57:43.612864 6347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:39065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:43.614305 6347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:43.615682 6347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:43.629062 6352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:43.629024 6355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:43.629135 6353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:43.629854 6347 server_base.cc:1034] running on GCE node
I20260926 02:57:43.630493 6347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:43.631515 6347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:43.632799 6347 hybrid_clock.cc:648] HybridClock initialized: now 1790391463632742 us; error 47 us; skew 500 ppm
I20260926 02:57:43.633141 6347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:43 dist-test-slave-ktng krb5kdc[5973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391463, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:43.636162 6347 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:57:43.637657 6347 webserver.cc:460] Webserver started at http://127.19.185.195:43731/ using document root <none> and password file <none>
I20260926 02:57:43.638324 6347 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:43.638456 6347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:43.638798 6347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:43.641836 6347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "91e766dfceca466db9660b694fa29f91"
format_stamp: "Formatted at 2026-09-26 02:57:43 on dist-test-slave-ktng"
server_key: "e0ab390a042e56475a49842dd1f3450c"
server_key_iv: "1e77c5c3231543d775196dcddaeb53e7"
server_key_version: "encryptionkey@0"
I20260926 02:57:43.642586 6347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "91e766dfceca466db9660b694fa29f91"
format_stamp: "Formatted at 2026-09-26 02:57:43 on dist-test-slave-ktng"
server_key: "e0ab390a042e56475a49842dd1f3450c"
server_key_iv: "1e77c5c3231543d775196dcddaeb53e7"
server_key_version: "encryptionkey@0"
I20260926 02:57:43.647753 6347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:57:43.651147 6362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:43.652541 6347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:57:43.652741 6347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "91e766dfceca466db9660b694fa29f91"
format_stamp: "Formatted at 2026-09-26 02:57:43 on dist-test-slave-ktng"
server_key: "e0ab390a042e56475a49842dd1f3450c"
server_key_iv: "1e77c5c3231543d775196dcddaeb53e7"
server_key_version: "encryptionkey@0"
I20260926 02:57:43.652937 6347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:43.665489 6347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:43.670317 6347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:43.670679 6347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:43.671801 6347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:43.673092 6347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:43.673197 6347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:43.673343 6347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:43.673416 6347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:43.696672 6207 heartbeater.cc:503] Master 127.19.185.254:39065 was elected leader, sending a full tablet report...
I20260926 02:57:43.708510 6347 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:43691
I20260926 02:57:43.708549 6474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:43691 every 8 connection(s)
I20260926 02:57:43.709872 6347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:57:43.711607 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 6347
I20260926 02:57:43.711788 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:57:43.712282 20199 external_mini_cluster.cc:1428] Setting key ca8113202e047c6d7063ae07fbd96f26
Sep 26 02:57:43 dist-test-slave-ktng krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391463, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:57:43.731350 6343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.712798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:53075 (local address 127.19.185.254:39065)
0926 02:57:43.713002 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:43.713008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:43.714421 (+ 1413us) server_negotiation.cc:407] Connection header received
0926 02:57:43.716433 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:43.716437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:43.716518 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:43.716602 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:43.718281 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.719033 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.720773 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.721179 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.721283 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.725036 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:43.725042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:43.725045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:43.725079 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:43.728262 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:43.729031 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:43.729041 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:43.729043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:43.729160 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:43.729595 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:43.729600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:43.729603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:43.729889 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:43.730042 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:43.730746 (+ 704us) server_negotiation.cc:299] Negotiation successful
0926 02:57:43.730984 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:57:43.732416 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.713386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:39065 (local address 127.19.185.195:53075)
0926 02:57:43.714266 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:43.714314 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:43.716168 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:43.716896 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:43.716905 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:43.717501 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:43.718107 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:43.718126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.719158 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.719162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:43.720612 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:43.720625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.721373 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.721377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:43.721589 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.722225 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:57:43.722248 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:43.724790 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:43.728413 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:43.728422 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:43.728441 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:43.728897 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:43.729270 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:43.729275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:43.729279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:43.729475 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:43.730017 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:43.730024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:43.730381 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 02:57:43.730642 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 02:57:43.731200 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":379,"threads_started":1}
I20260926 02:57:43.733968 6475 heartbeater.cc:347] Connected to a master server at 127.19.185.254:39065
I20260926 02:57:43.734248 6475 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:43.735001 6475 heartbeater.cc:511] Master 127.19.185.254:39065 requested a full tablet report, sending...
I20260926 02:57:43.736572 6018 ts_manager.cc:194] Registered new tserver with Master: 91e766dfceca466db9660b694fa29f91 (127.19.185.195:43691)
I20260926 02:57:43.737403 6018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:53075
I20260926 02:57:43.740197 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:57:43.757810 6343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.744276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.19.185.254:39065)
0926 02:57:43.744460 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:43.744468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:43.744925 (+ 457us) server_negotiation.cc:407] Connection header received
0926 02:57:43.745130 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:43.745134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:43.745192 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:43.745293 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:43.746746 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.747548 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.748874 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.749292 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.749430 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.752391 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:43.752396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:43.752399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:43.752428 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:43.754998 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:43.755784 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:43.755787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:43.755790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:43.755977 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:43.756420 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:43.756423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:43.756426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:43.756695 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:43.756944 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:43.757341 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 02:57:43.757501 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:57:43.761837 6018 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:57:43.768777 6018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:57:43.794195 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.781505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:33339 (local address 127.0.0.1:54114)
0926 02:57:43.785479 (+ 3974us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:43.785497 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:43.785619 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:43.786647 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:43.786652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:43.786678 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:43.787238 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:43.787253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.789207 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.789212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:43.791508 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:43.791521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.793140 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.793146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:43.793364 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.793413 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 02:57:43.793533 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 02:57:43.793719 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3847,"mutex_wait_us":54,"thread_start_us":3662,"threads_started":1}
I20260926 02:57:43.794482 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.781911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.19.185.194:33339)
0926 02:57:43.783663 (+ 1752us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:43.783676 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:43.785755 (+ 2079us) server_negotiation.cc:407] Connection header received
0926 02:57:43.785866 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:43.785876 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:43.786131 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:43.786361 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:43.787483 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.789040 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.791668 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.792648 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.792813 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.793002 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:43.793932 (+ 930us) server_negotiation.cc:299] Negotiation successful
0926 02:57:43.794127 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1523,"thread_start_us":1384,"threads_started":1}
I20260926 02:57:43.794822 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.781357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:43691 (local address 127.0.0.1:39942)
0926 02:57:43.784723 (+ 3366us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:43.784790 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:43.784990 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:43.785983 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:43.785992 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:43.786032 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:43.786476 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:43.786489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.788758 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.788763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:43.791431 (+ 2668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:43.791449 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.793775 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.793792 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:43.794032 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.794077 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 02:57:43.794175 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 02:57:43.794321 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3115,"mutex_wait_us":2,"thread_start_us":2947,"threads_started":1}
I20260926 02:57:43.795552 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.781630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.19.185.195:43691)
0926 02:57:43.784282 (+ 2652us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:43.784293 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:43.785017 (+ 724us) server_negotiation.cc:407] Connection header received
0926 02:57:43.785315 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:43.785326 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:43.785630 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:43.785967 (+ 337us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:43.786684 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.788596 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.791624 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.793223 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.793420 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.793615 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:43.794963 (+ 1348us) server_negotiation.cc:299] Negotiation successful
0926 02:57:43.795151 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2434,"thread_start_us":2290,"threads_started":1}
I20260926 02:57:43.795557 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.785927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.19.185.193:33949)
0926 02:57:43.786731 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:43.786741 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:43.787531 (+ 790us) server_negotiation.cc:407] Connection header received
0926 02:57:43.787746 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:43.787752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:43.787915 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:43.788119 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:43.788693 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.790181 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.792638 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.794035 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.794237 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.794429 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:43.795023 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 02:57:43.795184 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":429,"threads_started":1}
I20260926 02:57:43.796300 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.786680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:33949 (local address 127.0.0.1:54230)
0926 02:57:43.787405 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:43.787425 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:43.787552 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:43.788098 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:43.788103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:43.788130 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:43.788499 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:43.788514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.790334 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.790342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:43.792454 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:43.792469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.794575 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.794584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:43.794784 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.794833 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 02:57:43.795802 (+ 969us) client_negotiation.cc:240] Negotiation successful
0926 02:57:43.795940 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":11,"thread_start_us":319,"threads_started":1}
I20260926 02:57:43.798907 6276 tablet_service.cc:1467] Processing CreateTablet for tablet 1584294f70e741d191591adf6961663a (DEFAULT_TABLE table=test-table [id=589ea39588f944758de7df6f399ed08f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:43.799338 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 1584294f70e741d191591adf6961663a (DEFAULT_TABLE table=test-table [id=589ea39588f944758de7df6f399ed08f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:43.800276 6276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1584294f70e741d191591adf6961663a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:43.800033 6142 tablet_service.cc:1467] Processing CreateTablet for tablet 1584294f70e741d191591adf6961663a (DEFAULT_TABLE table=test-table [id=589ea39588f944758de7df6f399ed08f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:43.800858 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1584294f70e741d191591adf6961663a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:43.801501 6142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1584294f70e741d191591adf6961663a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:43.813396 6493 tablet_bootstrap.cc:492] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781: Bootstrap starting.
I20260926 02:57:43.816203 6495 tablet_bootstrap.cc:492] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91: Bootstrap starting.
I20260926 02:57:43.816982 6494 tablet_bootstrap.cc:492] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722: Bootstrap starting.
I20260926 02:57:43.817006 6493 tablet_bootstrap.cc:654] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:43.818387 6493 log.cc:824] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:43.820115 6495 tablet_bootstrap.cc:654] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:43.820523 6494 tablet_bootstrap.cc:654] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:43.821249 6495 log.cc:824] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:43.821813 6494 log.cc:824] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:43.822705 6493 tablet_bootstrap.cc:492] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781: No bootstrap required, opened a new log
I20260926 02:57:43.823266 6493 ts_tablet_manager.cc:1397] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 02:57:43.824470 6495 tablet_bootstrap.cc:492] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91: No bootstrap required, opened a new log
I20260926 02:57:43.824738 6495 ts_tablet_manager.cc:1397] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:57:43.825910 6494 tablet_bootstrap.cc:492] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722: No bootstrap required, opened a new log
I20260926 02:57:43.826154 6494 ts_tablet_manager.cc:1397] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260926 02:57:43.831732 6495 raft_consensus.cc:348] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e766dfceca466db9660b694fa29f91" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43691 } } peers { permanent_uuid: "250719e0e2394e2b877cf794ec2d2781" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 33339 } } peers { permanent_uuid: "258acff31d2e4794a59d690303211722" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 33949 } }
I20260926 02:57:43.832271 6495 raft_consensus.cc:374] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:43.832396 6495 raft_consensus.cc:729] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e766dfceca466db9660b694fa29f91, State: Initialized, Role: FOLLOWER
I20260926 02:57:43.833130 6495 consensus_queue.cc:260] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e766dfceca466db9660b694fa29f91" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43691 } } peers { permanent_uuid: "250719e0e2394e2b877cf794ec2d2781" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 33339 } } peers { permanent_uuid: "258acff31d2e4794a59d690303211722" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 33949 } }
I20260926 02:57:43.833997 6493 raft_consensus.cc:348] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e766dfceca466db9660b694fa29f91" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43691 } } peers { permanent_uuid: "250719e0e2394e2b877cf794ec2d2781" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 33339 } } peers { permanent_uuid: "258acff31d2e4794a59d690303211722" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 33949 } }
I20260926 02:57:43.834342 6475 heartbeater.cc:503] Master 127.19.185.254:39065 was elected leader, sending a full tablet report...
I20260926 02:57:43.834447 6493 raft_consensus.cc:374] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:43.834550 6493 raft_consensus.cc:729] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250719e0e2394e2b877cf794ec2d2781, State: Initialized, Role: FOLLOWER
I20260926 02:57:43.835069 6495 ts_tablet_manager.cc:1428] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260926 02:57:43.835155 6493 consensus_queue.cc:260] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e766dfceca466db9660b694fa29f91" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43691 } } peers { permanent_uuid: "250719e0e2394e2b877cf794ec2d2781" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 33339 } } peers { permanent_uuid: "258acff31d2e4794a59d690303211722" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 33949 } }
I20260926 02:57:43.836302 6341 heartbeater.cc:503] Master 127.19.185.254:39065 was elected leader, sending a full tablet report...
I20260926 02:57:43.836966 6493 ts_tablet_manager.cc:1428] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781: Time spent starting tablet: real 0.013s user 0.000s sys 0.013s
I20260926 02:57:43.837788 6494 raft_consensus.cc:348] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e766dfceca466db9660b694fa29f91" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43691 } } peers { permanent_uuid: "250719e0e2394e2b877cf794ec2d2781" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 33339 } } peers { permanent_uuid: "258acff31d2e4794a59d690303211722" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 33949 } }
I20260926 02:57:43.838203 6494 raft_consensus.cc:374] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:43.838325 6494 raft_consensus.cc:729] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 258acff31d2e4794a59d690303211722, State: Initialized, Role: FOLLOWER
I20260926 02:57:43.838900 6494 consensus_queue.cc:260] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e766dfceca466db9660b694fa29f91" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43691 } } peers { permanent_uuid: "250719e0e2394e2b877cf794ec2d2781" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 33339 } } peers { permanent_uuid: "258acff31d2e4794a59d690303211722" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 33949 } }
I20260926 02:57:43.840462 6494 ts_tablet_manager.cc:1428] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260926 02:57:43.911657 6208 tablet.cc:2369] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:57:43.931937 6500 raft_consensus.cc:482] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:57:43.932240 6500 raft_consensus.cc:504] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e766dfceca466db9660b694fa29f91" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43691 } } peers { permanent_uuid: "250719e0e2394e2b877cf794ec2d2781" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 33339 } } peers { permanent_uuid: "258acff31d2e4794a59d690303211722" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 33949 } }
I20260926 02:57:43.934000 6500 leader_election.cc:288] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91e766dfceca466db9660b694fa29f91 (127.19.185.195:43691), 258acff31d2e4794a59d690303211722 (127.19.185.193:33949)
I20260926 02:57:43.943622 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.934435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:39113 (local address 127.19.185.193:33949)
0926 02:57:43.934638 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:43.934644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:43.935519 (+ 875us) server_negotiation.cc:407] Connection header received
0926 02:57:43.935776 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:43.935784 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:43.935923 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:43.936051 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:43.936820 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.938059 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.941068 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.942411 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.942584 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.942659 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:43.943205 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 02:57:43.943369 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:57:43.943619 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.934435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:34179 (local address 127.19.185.195:43691)
0926 02:57:43.934638 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:43.934644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:43.935452 (+ 808us) server_negotiation.cc:407] Connection header received
0926 02:57:43.935839 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:43.935844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:43.935919 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:43.936014 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:43.936950 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.938059 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.941077 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.942443 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.942585 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.942659 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:43.943205 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 02:57:43.943369 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:57:43.943866 6503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.934344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:33949 (local address 127.19.185.194:39113)
0926 02:57:43.935321 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:43.935342 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:43.935596 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:43.936079 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:43.936083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:43.936111 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:43.936583 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:43.936598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.938182 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.938191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:43.940884 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:43.940900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.942716 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.942721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:43.942986 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.943055 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 02:57:43.943153 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 02:57:43.943361 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":178,"thread_start_us":536,"threads_started":1}
I20260926 02:57:43.943939 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.934289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:43691 (local address 127.19.185.194:34179)
0926 02:57:43.935321 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:43.935342 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:43.935612 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:43.936203 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:43.936208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:43.936248 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:43.936730 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:43.936747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.938198 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.938202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:43.940884 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:43.940900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.942710 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.942717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:43.942980 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.943052 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 02:57:43.943153 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 02:57:43.943361 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":200,"spinlock_wait_cycles":224640,"thread_start_us":579,"threads_started":1}
I20260926 02:57:43.944965 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1584294f70e741d191591adf6961663a" candidate_uuid: "250719e0e2394e2b877cf794ec2d2781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e766dfceca466db9660b694fa29f91" is_pre_election: true
I20260926 02:57:43.945245 6162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1584294f70e741d191591adf6961663a" candidate_uuid: "250719e0e2394e2b877cf794ec2d2781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "258acff31d2e4794a59d690303211722" is_pre_election: true
I20260926 02:57:43.945475 6430 raft_consensus.cc:2454] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 250719e0e2394e2b877cf794ec2d2781 in term 0.
I20260926 02:57:43.945788 6162 raft_consensus.cc:2454] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 250719e0e2394e2b877cf794ec2d2781 in term 0.
I20260926 02:57:43.946246 6231 leader_election.cc:302] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 250719e0e2394e2b877cf794ec2d2781, 91e766dfceca466db9660b694fa29f91; no voters:
I20260926 02:57:43.946707 6500 raft_consensus.cc:2781] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:57:43.946861 6500 raft_consensus.cc:482] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:57:43.946974 6500 raft_consensus.cc:3037] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:43.949170 6500 raft_consensus.cc:504] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e766dfceca466db9660b694fa29f91" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43691 } } peers { permanent_uuid: "250719e0e2394e2b877cf794ec2d2781" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 33339 } } peers { permanent_uuid: "258acff31d2e4794a59d690303211722" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 33949 } }
I20260926 02:57:43.949792 6500 leader_election.cc:288] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [CANDIDATE]: Term 1 election: Requested vote from peers 91e766dfceca466db9660b694fa29f91 (127.19.185.195:43691), 258acff31d2e4794a59d690303211722 (127.19.185.193:33949)
I20260926 02:57:43.950322 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1584294f70e741d191591adf6961663a" candidate_uuid: "250719e0e2394e2b877cf794ec2d2781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e766dfceca466db9660b694fa29f91"
I20260926 02:57:43.950350 6162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1584294f70e741d191591adf6961663a" candidate_uuid: "250719e0e2394e2b877cf794ec2d2781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "258acff31d2e4794a59d690303211722"
I20260926 02:57:43.950543 6430 raft_consensus.cc:3037] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:43.950556 6162 raft_consensus.cc:3037] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722 [term 0 FOLLOWER]: Advancing to term 1
W20260926 02:57:43.951956 6342 tablet.cc:2369] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:57:43.953017 6162 raft_consensus.cc:2454] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 250719e0e2394e2b877cf794ec2d2781 in term 1.
I20260926 02:57:43.953398 6430 raft_consensus.cc:2454] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 250719e0e2394e2b877cf794ec2d2781 in term 1.
I20260926 02:57:43.953454 6229 leader_election.cc:302] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 250719e0e2394e2b877cf794ec2d2781, 258acff31d2e4794a59d690303211722; no voters:
I20260926 02:57:43.953794 6500 raft_consensus.cc:2781] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:57:43.954161 6500 raft_consensus.cc:686] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [term 1 LEADER]: Becoming Leader. State: Replica: 250719e0e2394e2b877cf794ec2d2781, State: Running, Role: LEADER
I20260926 02:57:43.954607 6500 consensus_queue.cc:237] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e766dfceca466db9660b694fa29f91" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43691 } } peers { permanent_uuid: "250719e0e2394e2b877cf794ec2d2781" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 33339 } } peers { permanent_uuid: "258acff31d2e4794a59d690303211722" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 33949 } }
I20260926 02:57:43.959563 6016 catalog_manager.cc:5495] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 reported cstate change: term changed from 0 to 1, leader changed from <none> to 250719e0e2394e2b877cf794ec2d2781 (127.19.185.194). New cstate: current_term: 1 leader_uuid: "250719e0e2394e2b877cf794ec2d2781" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e766dfceca466db9660b694fa29f91" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "250719e0e2394e2b877cf794ec2d2781" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 33339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "258acff31d2e4794a59d690303211722" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 33949 } health_report { overall_health: UNKNOWN } } }
W20260926 02:57:43.961494 6476 tablet.cc:2369] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:57:43.986105 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:43.980634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.19.185.194:33339)
0926 02:57:43.980809 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:43.980814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:43.980830 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:57:43.980962 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:43.980966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:43.981049 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:43.981170 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:57:43.981565 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.982335 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.983832 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:43.984236 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:43.984379 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:43.984992 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:57:43.985489 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:57:43.985673 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:43.985816 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 02:57:43.985867 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:57:43.994686 6430 raft_consensus.cc:1261] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91 [term 1 FOLLOWER]: Refusing update from remote peer 250719e0e2394e2b877cf794ec2d2781: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:57:43.994685 6162 raft_consensus.cc:1261] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722 [term 1 FOLLOWER]: Refusing update from remote peer 250719e0e2394e2b877cf794ec2d2781: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:57:43.995877 6504 consensus_queue.cc:1035] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [LEADER]: Connected to new peer: Peer: permanent_uuid: "258acff31d2e4794a59d690303211722" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 33949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:57:43.996248 6500 consensus_queue.cc:1035] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e766dfceca466db9660b694fa29f91" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:57:44.026449 6018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:57:44.026754 6018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:57:44.030474 6018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11ad29df20de41ffb83cc030ce22a54a: Sending DeleteTablet for 3 replicas of tablet 1584294f70e741d191591adf6961663a
I20260926 02:57:44.031585 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 1584294f70e741d191591adf6961663a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:44 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:39942
I20260926 02:57:44.031730 6276 tablet_service.cc:1514] Processing DeleteTablet for tablet 1584294f70e741d191591adf6961663a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:44 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:54114
I20260926 02:57:44.031971 6142 tablet_service.cc:1514] Processing DeleteTablet for tablet 1584294f70e741d191591adf6961663a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:44 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:54230
I20260926 02:57:44.032518 6517 tablet_replica.cc:330] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91: stopping tablet replica
I20260926 02:57:44.032955 6517 raft_consensus.cc:2229] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:57:44.033555 6517 raft_consensus.cc:2258] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:44.033798 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 6079
I20260926 02:57:44.041915 6517 ts_tablet_manager.cc:1905] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:44.049882 6518 tablet_replica.cc:330] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781: stopping tablet replica
I20260926 02:57:44.050459 6518 raft_consensus.cc:2229] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:57:44.051198 6518 raft_consensus.cc:2258] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:44.053613 6518 ts_tablet_manager.cc:1905] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:44.055861 6519 tablet_replica.cc:330] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722: stopping tablet replica
I20260926 02:57:44.056363 6519 raft_consensus.cc:2229] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:57:44.057014 6519 raft_consensus.cc:2258] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:44.059386 6519 ts_tablet_manager.cc:1905] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:44.061806 6518 ts_tablet_manager.cc:1918] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:44.062058 6518 log.cc:1196] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/1584294f70e741d191591adf6961663a
I20260926 02:57:44.062696 6518 ts_tablet_manager.cc:1939] T 1584294f70e741d191591adf6961663a P 250719e0e2394e2b877cf794ec2d2781: Deleting consensus metadata
I20260926 02:57:44.064775 6002 catalog_manager.cc:4841] TS 250719e0e2394e2b877cf794ec2d2781 (127.19.185.194:33339): tablet 1584294f70e741d191591adf6961663a (table test-table [id=589ea39588f944758de7df6f399ed08f]) successfully deleted
Sep 26 02:57:44 dist-test-slave-ktng krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:57:44.066473 6519 ts_tablet_manager.cc:1918] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:44.066730 6519 log.cc:1196] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/1584294f70e741d191591adf6961663a
I20260926 02:57:44.067363 6519 ts_tablet_manager.cc:1939] T 1584294f70e741d191591adf6961663a P 258acff31d2e4794a59d690303211722: Deleting consensus metadata
I20260926 02:57:44.068002 6517 ts_tablet_manager.cc:1918] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:44.068222 6517 log.cc:1196] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/1584294f70e741d191591adf6961663a
I20260926 02:57:44.068794 6517 ts_tablet_manager.cc:1939] T 1584294f70e741d191591adf6961663a P 91e766dfceca466db9660b694fa29f91: Deleting consensus metadata
I20260926 02:57:44.069546 6002 catalog_manager.cc:4841] TS 258acff31d2e4794a59d690303211722 (127.19.185.193:33949): tablet 1584294f70e741d191591adf6961663a (table test-table [id=589ea39588f944758de7df6f399ed08f]) successfully deleted
I20260926 02:57:44.070510 6004 catalog_manager.cc:4841] TS 91e766dfceca466db9660b694fa29f91 (127.19.185.195:43691): tablet 1584294f70e741d191591adf6961663a (table test-table [id=589ea39588f944758de7df6f399ed08f]) successfully deleted
I20260926 02:57:44.074402 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:44.034981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.19.185.193:33949)
0926 02:57:44.035411 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:44.035420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:44.054719 (+ 19299us) server_negotiation.cc:407] Connection header received
0926 02:57:44.054916 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:44.054924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:44.055021 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:44.055145 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:44.057127 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:44.058471 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:44.060607 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:44.061297 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:44.061438 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:44.067273 (+ 5835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:44.067311 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:44.067331 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:44.067397 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:44.071380 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:44.072093 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:44.072103 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:44.072107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:44.072322 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:44.072694 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:44.072700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:44.072703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:44.073138 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:44.073461 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:44.073759 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 02:57:44.074002 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260926 02:57:44.075071 6202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:44.214193 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 6079
I20260926 02:57:44.232734 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 6213
Sep 26 02:57:44 dist-test-slave-ktng krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:57:44.252791 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:44.233767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54122 (local address 127.19.185.194:33339)
0926 02:57:44.233963 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:44.233967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:44.233999 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:57:44.234052 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:44.234055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:44.234112 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:44.234191 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:44.235897 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:44.236804 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:44.238750 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:44.239467 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:44.239572 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:44.245659 (+ 6087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:44.245679 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:44.245702 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:44.245750 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:44.249752 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:44.250448 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:44.250458 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:44.250462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:44.250634 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:44.251043 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:44.251049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:44.251053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:44.251577 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:44.251921 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:44.252157 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 02:57:44.252406 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:57:44.253785 6336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:44.368675 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 6213
I20260926 02:57:44.387760 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 6347
Sep 26 02:57:44 dist-test-slave-ktng krb5kdc[5973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:57:44.402935 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:44.388570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.19.185.195:43691)
0926 02:57:44.388757 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:44.388764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:44.388791 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:57:44.389021 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:44.389025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:44.389088 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:44.389198 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:44.390481 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:44.391190 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:44.392499 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:44.392897 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:44.393034 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:44.397677 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:44.397704 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:44.397720 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:44.397764 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:44.400560 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:44.401260 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:44.401267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:44.401269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:44.401406 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:44.401706 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:44.401709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:44.401711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:44.402041 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:44.402253 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:44.402489 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 02:57:44.402643 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:57:44.403705 6470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:44.492847 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 6347
I20260926 02:57:44.510020 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 5986
I20260926 02:57:44.511145 6048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:44.623479 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 5986
2026-09-26T02:57:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4478 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T02:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:57:44Z Disabled control of system clock
I20260926 02:57:44.703056 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:35349
--webserver_interface=127.19.185.254
--webserver_port=0
--builtin_ntp_servers=127.19.185.212:38045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:35349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:57:45.035022 6528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:45.035391 6528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:45.035540 6528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:45.042812 6528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:57:45.042965 6528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:45.043032 6528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:45.043118 6528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:57:45.043201 6528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:57:45.052498 6528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:38045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:35349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:35349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:45.053881 6528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:45.055760 6528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:45.067797 6534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:45.067770 6533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:45.068476 6536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:45.068876 6528 server_base.cc:1034] running on GCE node
I20260926 02:57:45.069576 6528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:45.070931 6528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:45.072167 6528 hybrid_clock.cc:648] HybridClock initialized: now 1790391465072129 us; error 49 us; skew 500 ppm
I20260926 02:57:45.072556 6528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:57:45.074770 6528 webserver.cc:460] Webserver started at http://127.19.185.254:37695/ using document root <none> and password file <none>
I20260926 02:57:45.075574 6528 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:45.075707 6528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:45.076033 6528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:45.078922 6528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "e066f9151c78409da87262ecc220dfa2"
format_stamp: "Formatted at 2026-09-26 02:57:45 on dist-test-slave-ktng"
server_key: "1c816177ad6d496a87a9867d52de6d61"
server_key_iv: "7102bf5a37b1e6c429ba4e7459bdb788"
server_key_version: "encryptionkey@0"
I20260926 02:57:45.079759 6528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "e066f9151c78409da87262ecc220dfa2"
format_stamp: "Formatted at 2026-09-26 02:57:45 on dist-test-slave-ktng"
server_key: "1c816177ad6d496a87a9867d52de6d61"
server_key_iv: "7102bf5a37b1e6c429ba4e7459bdb788"
server_key_version: "encryptionkey@0"
I20260926 02:57:45.084936 6528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:57:45.088442 6542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:45.089879 6528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:57:45.090090 6528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "e066f9151c78409da87262ecc220dfa2"
format_stamp: "Formatted at 2026-09-26 02:57:45 on dist-test-slave-ktng"
server_key: "1c816177ad6d496a87a9867d52de6d61"
server_key_iv: "7102bf5a37b1e6c429ba4e7459bdb788"
server_key_version: "encryptionkey@0"
I20260926 02:57:45.090325 6528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:45.105269 6528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:45.106087 6528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:45.106391 6528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:45.125288 6528 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:35349
I20260926 02:57:45.125325 6593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:35349 every 8 connection(s)
I20260926 02:57:45.126678 6528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:57:45.130512 6594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:45.135677 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 6528
I20260926 02:57:45.135869 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:57:45.136355 20199 external_mini_cluster.cc:1428] Setting key 36ab4b5d87476340ad83ac5778f4474b
I20260926 02:57:45.141682 6594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2: Bootstrap starting.
I20260926 02:57:45.144814 6594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:45.145841 6594 log.cc:824] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:45.149392 6594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2: No bootstrap required, opened a new log
I20260926 02:57:45.150637 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:45.138389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.19.185.254:35349)
0926 02:57:45.139553 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:45.139565 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:45.139608 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:57:45.141283 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:45.141312 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:45.141886 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:45.142270 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:45.142827 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:45.144208 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:45.145600 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:45.146338 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:45.146554 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:45.147813 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:45.147823 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:45.147840 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:45.148570 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:45.148684 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:45.148725 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:45.148843 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:45.149067 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 02:57:45.149392 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":420,"threads_started":1}
I20260926 02:57:45.157805 6594 raft_consensus.cc:348] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e066f9151c78409da87262ecc220dfa2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 35349 } }
I20260926 02:57:45.158179 6594 raft_consensus.cc:374] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:45.158293 6594 raft_consensus.cc:729] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e066f9151c78409da87262ecc220dfa2, State: Initialized, Role: FOLLOWER
I20260926 02:57:45.158967 6594 consensus_queue.cc:260] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e066f9151c78409da87262ecc220dfa2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 35349 } }
I20260926 02:57:45.159212 6594 raft_consensus.cc:388] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:57:45.159356 6594 raft_consensus.cc:482] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:57:45.159534 6594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:45.163123 6594 raft_consensus.cc:504] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e066f9151c78409da87262ecc220dfa2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 35349 } }
I20260926 02:57:45.163818 6594 leader_election.cc:302] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e066f9151c78409da87262ecc220dfa2; no voters:
I20260926 02:57:45.164539 6594 leader_election.cc:288] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:57:45.164665 6599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:57:45.166749 6599 raft_consensus.cc:686] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [term 1 LEADER]: Becoming Leader. State: Replica: e066f9151c78409da87262ecc220dfa2, State: Running, Role: LEADER
I20260926 02:57:45.167447 6599 consensus_queue.cc:237] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e066f9151c78409da87262ecc220dfa2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 35349 } }
I20260926 02:57:45.167698 6594 sys_catalog.cc:564] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:57:45.170787 6601 sys_catalog.cc:455] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e066f9151c78409da87262ecc220dfa2. Latest consensus state: current_term: 1 leader_uuid: "e066f9151c78409da87262ecc220dfa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e066f9151c78409da87262ecc220dfa2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 35349 } } }
I20260926 02:57:45.170838 6600 sys_catalog.cc:455] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e066f9151c78409da87262ecc220dfa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e066f9151c78409da87262ecc220dfa2" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 35349 } } }
I20260926 02:57:45.171239 6601 sys_catalog.cc:458] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:45.171244 6600 sys_catalog.cc:458] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2 [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:45.173652 6609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:57:45.178171 6609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:57:45.187695 6609 catalog_manager.cc:1347] Generated new cluster ID: 770fef4cf3a5426eaa2c498587a92c02
I20260926 02:57:45.187886 6609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:57:45.205593 6609 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:57:45.206985 6609 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:57:45.229918 6609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2: Generated new TSK 0
I20260926 02:57:45.230808 6609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:57:45.237350 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:35349
--builtin_ntp_servers=127.19.185.212:38045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:57:45.574772 6618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:45.575057 6618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:45.575222 6618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:45.582742 6618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:45.582844 6618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:45.583035 6618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:57:45.592470 6618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:38045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:35349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:45.593900 6618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:45.595893 6618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:45.609241 6623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:45.609292 6626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:45.609202 6624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:45.610229 6618 server_base.cc:1034] running on GCE node
I20260926 02:57:45.610932 6618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:45.611938 6618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:45.613173 6618 hybrid_clock.cc:648] HybridClock initialized: now 1790391465613104 us; error 63 us; skew 500 ppm
I20260926 02:57:45.613479 6618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:57:45.623013 6618 webserver.cc:460] Webserver started at http://127.19.185.193:44049/ using document root <none> and password file <none>
I20260926 02:57:45.623818 6618 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:45.623924 6618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:45.624181 6618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:45.627130 6618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f"
format_stamp: "Formatted at 2026-09-26 02:57:45 on dist-test-slave-ktng"
server_key: "6b7f416126dbec241148bb931b0cff30"
server_key_iv: "8ebab3a5bce3fe6e9e839248830e8992"
server_key_version: "encryptionkey@0"
I20260926 02:57:45.627961 6618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f"
format_stamp: "Formatted at 2026-09-26 02:57:45 on dist-test-slave-ktng"
server_key: "6b7f416126dbec241148bb931b0cff30"
server_key_iv: "8ebab3a5bce3fe6e9e839248830e8992"
server_key_version: "encryptionkey@0"
I20260926 02:57:45.633075 6618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:57:45.636505 6632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:45.637952 6618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:57:45.638093 6618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f"
format_stamp: "Formatted at 2026-09-26 02:57:45 on dist-test-slave-ktng"
server_key: "6b7f416126dbec241148bb931b0cff30"
server_key_iv: "8ebab3a5bce3fe6e9e839248830e8992"
server_key_version: "encryptionkey@0"
I20260926 02:57:45.638275 6618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:45.653209 6618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:45.653977 6618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:45.654232 6618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:45.655144 6618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:45.656623 6618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:45.656709 6618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:45.656800 6618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:45.656857 6618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:45.691428 6618 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:46067
I20260926 02:57:45.691488 6744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:46067 every 8 connection(s)
I20260926 02:57:45.692780 6618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:57:45.702560 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 6618
I20260926 02:57:45.702754 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:57:45.703368 20199 external_mini_cluster.cc:1428] Setting key 41556b4b0cf1c60e3b6291b93126d51a
I20260926 02:57:45.706243 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:45.695667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:34787 (local address 127.19.185.254:35349)
0926 02:57:45.696117 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:45.696124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:45.697629 (+ 1505us) server_negotiation.cc:407] Connection header received
0926 02:57:45.699199 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:45.699203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:45.699279 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:45.699460 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:45.700525 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:45.701268 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:45.703179 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:45.703709 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:45.703809 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:45.705395 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:45.705402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:45.705407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:45.705428 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:45.705481 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:45.705505 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:45.705576 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:45.705909 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 02:57:45.705987 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":203,"threads_started":1}
I20260926 02:57:45.706537 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:35349
--builtin_ntp_servers=127.19.185.212:38045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:57:45.707764 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:45.696329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:35349 (local address 127.19.185.193:34787)
0926 02:57:45.697276 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:45.697312 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:45.698921 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:45.699665 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:45.699673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:45.699760 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:45.700342 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:45.700360 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:45.701402 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:45.701405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:45.703033 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:45.703051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:45.703945 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:45.703961 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:45.704210 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:45.705017 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:57:45.705019 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:45.705085 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:57:45.705195 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:45.705607 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:45.705619 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:45.705627 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 02:57:45.705822 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 02:57:45.706233 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":268,"threads_started":1}
I20260926 02:57:45.709296 6745 heartbeater.cc:347] Connected to a master server at 127.19.185.254:35349
I20260926 02:57:45.709623 6745 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:45.710561 6745 heartbeater.cc:511] Master 127.19.185.254:35349 requested a full tablet report, sending...
I20260926 02:57:45.712855 6559 ts_manager.cc:194] Registered new tserver with Master: d1f8f0d8099f417bbc9ccd8e7aee2a7f (127.19.185.193:46067)
I20260926 02:57:45.714397 6559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.185.193:34787
W20260926 02:57:46.037981 6749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:46.038267 6749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:46.038353 6749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:46.045760 6749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:46.045863 6749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:46.045974 6749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:57:46.055001 6749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:38045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:35349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:46.056437 6749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:46.058240 6749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:46.072191 6754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:46.072158 6755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:46.072252 6757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:46.072952 6749 server_base.cc:1034] running on GCE node
I20260926 02:57:46.073565 6749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:46.074604 6749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:46.075845 6749 hybrid_clock.cc:648] HybridClock initialized: now 1790391466075793 us; error 31 us; skew 500 ppm
I20260926 02:57:46.076174 6749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:57:46.078574 6749 webserver.cc:460] Webserver started at http://127.19.185.194:35983/ using document root <none> and password file <none>
I20260926 02:57:46.079274 6749 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:46.079464 6749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:46.079788 6749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:46.082697 6749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "b13bd37d0e20418d9a3153f75e4f27a2"
format_stamp: "Formatted at 2026-09-26 02:57:46 on dist-test-slave-ktng"
server_key: "547dc42b75d12f320eaccd20f4e87c75"
server_key_iv: "542a6ac2f7eeb32909f297d8311b4064"
server_key_version: "encryptionkey@0"
I20260926 02:57:46.083580 6749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "b13bd37d0e20418d9a3153f75e4f27a2"
format_stamp: "Formatted at 2026-09-26 02:57:46 on dist-test-slave-ktng"
server_key: "547dc42b75d12f320eaccd20f4e87c75"
server_key_iv: "542a6ac2f7eeb32909f297d8311b4064"
server_key_version: "encryptionkey@0"
I20260926 02:57:46.088902 6749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 02:57:46.092767 6763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:46.094310 6749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:57:46.094497 6749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "b13bd37d0e20418d9a3153f75e4f27a2"
format_stamp: "Formatted at 2026-09-26 02:57:46 on dist-test-slave-ktng"
server_key: "547dc42b75d12f320eaccd20f4e87c75"
server_key_iv: "542a6ac2f7eeb32909f297d8311b4064"
server_key_version: "encryptionkey@0"
I20260926 02:57:46.094719 6749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:46.131338 6749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:46.132354 6749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:46.132666 6749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:46.133867 6749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:46.135705 6749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:46.135808 6749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:46.135953 6749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:46.136036 6749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:46.173174 6749 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:38885
I20260926 02:57:46.173326 6875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:38885 every 8 connection(s)
I20260926 02:57:46.174557 6749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:57:46.183615 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 6749
I20260926 02:57:46.183881 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:57:46.184350 20199 external_mini_cluster.cc:1428] Setting key 7e57ee015ffb05182486e70adec2565f
I20260926 02:57:46.187376 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:35349
--builtin_ntp_servers=127.19.185.212:38045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:57:46.187736 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.177442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:48497 (local address 127.19.185.254:35349)
0926 02:57:46.177632 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:46.177637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:46.179189 (+ 1552us) server_negotiation.cc:407] Connection header received
0926 02:57:46.181037 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:46.181043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:46.181134 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:46.181246 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:46.182324 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.183037 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.184842 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.185276 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.185449 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.186901 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:46.186906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:46.186909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:46.186926 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:46.186957 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:46.186969 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:46.187071 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:46.187323 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 02:57:46.187423 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:57:46.188763 6878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.178064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:35349 (local address 127.19.185.194:48497)
0926 02:57:46.179030 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:46.179073 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:46.180748 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:46.181482 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:46.181491 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:46.181579 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:46.182150 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:46.182165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.183185 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.183192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:46.184695 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:46.184708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.185534 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.185541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:46.185771 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.186519 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:57:46.186522 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:46.186569 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:57:46.186657 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:46.187070 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:46.187083 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:46.187093 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:57:46.187287 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 02:57:46.187722 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":286,"threads_started":1}
I20260926 02:57:46.190387 6876 heartbeater.cc:347] Connected to a master server at 127.19.185.254:35349
I20260926 02:57:46.190680 6876 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:46.191444 6876 heartbeater.cc:511] Master 127.19.185.254:35349 requested a full tablet report, sending...
I20260926 02:57:46.192973 6559 ts_manager.cc:194] Registered new tserver with Master: b13bd37d0e20418d9a3153f75e4f27a2 (127.19.185.194:38885)
I20260926 02:57:46.194397 6559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.185.194:48497
W20260926 02:57:46.520753 6879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:46.521049 6879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:46.521134 6879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:46.528517 6879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:46.528612 6879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:46.528717 6879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:57:46.537729 6879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:38045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:35349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:46.539036 6879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:46.540827 6879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:46.553994 6884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:46.553951 6885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:46.553906 6887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:46.554826 6879 server_base.cc:1034] running on GCE node
I20260926 02:57:46.555513 6879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:46.556524 6879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:46.557734 6879 hybrid_clock.cc:648] HybridClock initialized: now 1790391466557698 us; error 39 us; skew 500 ppm
I20260926 02:57:46.558081 6879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:57:46.560439 6879 webserver.cc:460] Webserver started at http://127.19.185.195:38853/ using document root <none> and password file <none>
I20260926 02:57:46.561131 6879 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:46.561249 6879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:46.561587 6879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:46.564453 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "0e02674f4d4240a28530920141a03a5d"
format_stamp: "Formatted at 2026-09-26 02:57:46 on dist-test-slave-ktng"
server_key: "a21e1297416d1c68ba68287bb88c81ad"
server_key_iv: "32831b105c5421b3719f4c6cc1c7fcc1"
server_key_version: "encryptionkey@0"
I20260926 02:57:46.565227 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "0e02674f4d4240a28530920141a03a5d"
format_stamp: "Formatted at 2026-09-26 02:57:46 on dist-test-slave-ktng"
server_key: "a21e1297416d1c68ba68287bb88c81ad"
server_key_iv: "32831b105c5421b3719f4c6cc1c7fcc1"
server_key_version: "encryptionkey@0"
I20260926 02:57:46.570581 6879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:57:46.574258 6893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:46.575625 6879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:57:46.575799 6879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "0e02674f4d4240a28530920141a03a5d"
format_stamp: "Formatted at 2026-09-26 02:57:46 on dist-test-slave-ktng"
server_key: "a21e1297416d1c68ba68287bb88c81ad"
server_key_iv: "32831b105c5421b3719f4c6cc1c7fcc1"
server_key_version: "encryptionkey@0"
I20260926 02:57:46.576025 6879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:46.620210 6879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:46.621124 6879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:46.621445 6879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:46.622493 6879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:46.624099 6879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:46.624225 6879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:46.624368 6879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:46.624461 6879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:46.658078 6879 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:40103
I20260926 02:57:46.658133 7005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:40103 every 8 connection(s)
I20260926 02:57:46.659476 6879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:57:46.665912 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 6879
I20260926 02:57:46.666109 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:57:46.666577 20199 external_mini_cluster.cc:1428] Setting key 883438bd6b4736429042025192a6ab87
I20260926 02:57:46.672874 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.662108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:36483 (local address 127.19.185.254:35349)
0926 02:57:46.662318 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:46.662323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:46.663941 (+ 1618us) server_negotiation.cc:407] Connection header received
0926 02:57:46.665701 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:46.665706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:46.665796 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:46.665943 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:46.667205 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.668160 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.669954 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.670466 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.670592 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.672023 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:46.672029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:46.672034 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:46.672059 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:46.672091 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:46.672103 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:46.672200 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:46.672427 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 02:57:46.672545 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 02:57:46.673826 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.662710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:35349 (local address 127.19.185.195:36483)
0926 02:57:46.663779 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:46.663815 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:46.665415 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:46.666201 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:46.666214 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:46.666315 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:46.667038 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:46.667057 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.668320 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.668325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:46.669808 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:46.669820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.670684 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.670691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:46.670918 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.671619 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:57:46.671622 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:46.671666 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:57:46.671743 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:46.672197 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:46.672209 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:46.672219 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:57:46.672419 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 02:57:46.672804 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":291,"threads_started":1}
I20260926 02:57:46.675087 7006 heartbeater.cc:347] Connected to a master server at 127.19.185.254:35349
I20260926 02:57:46.675408 7006 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:46.676108 7006 heartbeater.cc:511] Master 127.19.185.254:35349 requested a full tablet report, sending...
I20260926 02:57:46.677459 6559 ts_manager.cc:194] Registered new tserver with Master: 0e02674f4d4240a28530920141a03a5d (127.19.185.195:40103)
I20260926 02:57:46.678241 6559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.185.195:36483
I20260926 02:57:46.682826 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:57:46.693545 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.687029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.19.185.254:35349)
0926 02:57:46.687190 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:46.687194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:46.687600 (+ 406us) server_negotiation.cc:407] Connection header received
0926 02:57:46.687824 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:46.687829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:46.687893 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:46.687984 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:46.688484 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.689417 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.690828 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.691508 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.691646 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.692742 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:46.692748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:46.692753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:46.692776 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:46.692818 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:46.692835 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:46.692991 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:46.693159 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 02:57:46.693227 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:57:46.697546 6559 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:57:46.704329 6559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:57:46.718133 6745 heartbeater.cc:503] Master 127.19.185.254:35349 was elected leader, sending a full tablet report...
I20260926 02:57:46.728281 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.715897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:38885 (local address 127.0.0.1:37540)
0926 02:57:46.717457 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:46.717499 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:46.717713 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:46.718763 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:46.718772 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:46.718815 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:46.719361 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:46.719384 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.721573 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.721581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:46.724374 (+ 2793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:46.724395 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.727102 (+ 2707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.727109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:46.727439 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.727503 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 02:57:46.727634 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 02:57:46.727861 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1245,"mutex_wait_us":26,"thread_start_us":467,"threads_started":1}
I20260926 02:57:46.728969 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.716212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.19.185.194:38885)
0926 02:57:46.716923 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:46.716932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:46.717940 (+ 1008us) server_negotiation.cc:407] Connection header received
0926 02:57:46.718047 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:46.718054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:46.718305 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:46.718492 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:46.719568 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.721366 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.724674 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.726520 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.726689 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.726947 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:46.728356 (+ 1409us) server_negotiation.cc:299] Negotiation successful
0926 02:57:46.728593 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":420,"threads_started":1}
I20260926 02:57:46.732883 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.717226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:40103 (local address 127.0.0.1:55404)
0926 02:57:46.717806 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:46.717822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:46.717926 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:46.725869 (+ 7943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:46.725875 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:46.725912 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:46.726278 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:46.726288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.728438 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.728443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:46.730457 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:46.730469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.732085 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.732091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:46.732265 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.732300 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:57:46.732384 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 02:57:46.732526 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":2,"thread_start_us":297,"threads_started":1}
I20260926 02:57:46.733096 6811 tablet_service.cc:1467] Processing CreateTablet for tablet 995c3f8358c145b79903fa8c2761ccd8 (DEFAULT_TABLE table=test-table [id=99996abfda5c4508baeb881a2afa5de8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:46.734174 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.717574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.19.185.195:40103)
0926 02:57:46.725067 (+ 7493us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:46.725080 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:46.725124 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:57:46.725240 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:46.725252 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:46.725584 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:46.726452 (+ 868us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:46.726659 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.728279 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.730643 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.731876 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.733337 (+ 1461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.733576 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:46.733643 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 02:57:46.733813 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7250,"thread_start_us":7617,"threads_started":1}
I20260926 02:57:46.734277 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.717876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46067 (local address 127.0.0.1:36948)
0926 02:57:46.724574 (+ 6698us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:46.724592 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:46.724711 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:46.726849 (+ 2138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:46.726854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:46.726881 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:46.727355 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:46.727369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.729375 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.729383 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:46.731255 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:46.731270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.733503 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.733510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:46.733683 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.733715 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:57:46.733790 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 02:57:46.733934 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6571,"mutex_wait_us":9,"thread_start_us":6420,"threads_started":1}
I20260926 02:57:46.734906 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.718503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.19.185.193:46067)
0926 02:57:46.726246 (+ 7743us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:46.726255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:46.726283 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:57:46.726390 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:46.726397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:46.726642 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:46.726832 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:46.727678 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.729223 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.731487 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.732941 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.733140 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.733374 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:46.734495 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0926 02:57:46.734650 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7571,"thread_start_us":7445,"threads_started":1}
I20260926 02:57:46.735509 6811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995c3f8358c145b79903fa8c2761ccd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:46.738029 6941 tablet_service.cc:1467] Processing CreateTablet for tablet 995c3f8358c145b79903fa8c2761ccd8 (DEFAULT_TABLE table=test-table [id=99996abfda5c4508baeb881a2afa5de8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:46.738376 6680 tablet_service.cc:1467] Processing CreateTablet for tablet 995c3f8358c145b79903fa8c2761ccd8 (DEFAULT_TABLE table=test-table [id=99996abfda5c4508baeb881a2afa5de8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:46.739723 6941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995c3f8358c145b79903fa8c2761ccd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:46.739723 6680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995c3f8358c145b79903fa8c2761ccd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:46.749402 7024 tablet_bootstrap.cc:492] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2: Bootstrap starting.
I20260926 02:57:46.753031 7024 tablet_bootstrap.cc:654] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:46.754072 7024 log.cc:824] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:46.754997 7025 tablet_bootstrap.cc:492] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f: Bootstrap starting.
I20260926 02:57:46.755862 7026 tablet_bootstrap.cc:492] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d: Bootstrap starting.
I20260926 02:57:46.758673 7024 tablet_bootstrap.cc:492] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2: No bootstrap required, opened a new log
I20260926 02:57:46.758963 7024 ts_tablet_manager.cc:1397] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260926 02:57:46.759724 7026 tablet_bootstrap.cc:654] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:46.760097 7025 tablet_bootstrap.cc:654] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:46.761072 7026 log.cc:824] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:46.761229 7025 log.cc:824] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:46.764050 7026 tablet_bootstrap.cc:492] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d: No bootstrap required, opened a new log
I20260926 02:57:46.764323 7026 ts_tablet_manager.cc:1397] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260926 02:57:46.764467 7025 tablet_bootstrap.cc:492] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f: No bootstrap required, opened a new log
I20260926 02:57:46.764719 7025 ts_tablet_manager.cc:1397] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 02:57:46.768951 7024 raft_consensus.cc:348] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38885 } } peers { permanent_uuid: "0e02674f4d4240a28530920141a03a5d" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40103 } } peers { permanent_uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46067 } }
I20260926 02:57:46.769472 7024 raft_consensus.cc:374] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:46.769600 7024 raft_consensus.cc:729] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13bd37d0e20418d9a3153f75e4f27a2, State: Initialized, Role: FOLLOWER
I20260926 02:57:46.770290 7024 consensus_queue.cc:260] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38885 } } peers { permanent_uuid: "0e02674f4d4240a28530920141a03a5d" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40103 } } peers { permanent_uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46067 } }
I20260926 02:57:46.771497 6876 heartbeater.cc:503] Master 127.19.185.254:35349 was elected leader, sending a full tablet report...
I20260926 02:57:46.772220 7024 ts_tablet_manager.cc:1428] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260926 02:57:46.773360 7026 raft_consensus.cc:348] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38885 } } peers { permanent_uuid: "0e02674f4d4240a28530920141a03a5d" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40103 } } peers { permanent_uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46067 } }
I20260926 02:57:46.773708 7026 raft_consensus.cc:374] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:46.773864 7026 raft_consensus.cc:729] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e02674f4d4240a28530920141a03a5d, State: Initialized, Role: FOLLOWER
I20260926 02:57:46.774444 7026 consensus_queue.cc:260] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38885 } } peers { permanent_uuid: "0e02674f4d4240a28530920141a03a5d" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40103 } } peers { permanent_uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46067 } }
I20260926 02:57:46.774663 7025 raft_consensus.cc:348] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38885 } } peers { permanent_uuid: "0e02674f4d4240a28530920141a03a5d" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40103 } } peers { permanent_uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46067 } }
I20260926 02:57:46.775149 7025 raft_consensus.cc:374] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:46.775339 7025 raft_consensus.cc:729] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f8f0d8099f417bbc9ccd8e7aee2a7f, State: Initialized, Role: FOLLOWER
I20260926 02:57:46.775517 7006 heartbeater.cc:503] Master 127.19.185.254:35349 was elected leader, sending a full tablet report...
I20260926 02:57:46.775954 7025 consensus_queue.cc:260] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38885 } } peers { permanent_uuid: "0e02674f4d4240a28530920141a03a5d" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40103 } } peers { permanent_uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46067 } }
I20260926 02:57:46.776201 7026 ts_tablet_manager.cc:1428] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260926 02:57:46.778435 7025 ts_tablet_manager.cc:1428] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 02:57:46.829089 7030 raft_consensus.cc:482] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:57:46.829413 7030 raft_consensus.cc:504] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38885 } } peers { permanent_uuid: "0e02674f4d4240a28530920141a03a5d" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40103 } } peers { permanent_uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46067 } }
I20260926 02:57:46.831105 7030 leader_election.cc:288] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e02674f4d4240a28530920141a03a5d (127.19.185.195:40103), d1f8f0d8099f417bbc9ccd8e7aee2a7f (127.19.185.193:46067)
I20260926 02:57:46.837577 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.831575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:46299 (local address 127.19.185.195:40103)
0926 02:57:46.831820 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:46.831829 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:46.832336 (+ 507us) server_negotiation.cc:407] Connection header received
0926 02:57:46.832576 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:46.832584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:46.832684 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:46.832783 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:46.833391 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.834099 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.835911 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.836695 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.836888 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.836963 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:46.837216 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 02:57:46.837338 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:57:46.837649 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.831542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:40103 (local address 127.19.185.194:46299)
0926 02:57:46.832183 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:46.832199 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:46.832396 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:46.832842 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:46.832847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:46.832872 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:46.833241 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:46.833250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.834228 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.834236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:46.835713 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:46.835730 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.836889 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.836895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:46.837059 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.837082 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:57:46.837185 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 02:57:46.837321 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":384,"threads_started":1}
I20260926 02:57:46.838048 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.832491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:52553 (local address 127.19.185.193:46067)
0926 02:57:46.832707 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:46.832713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:46.833039 (+ 326us) server_negotiation.cc:407] Connection header received
0926 02:57:46.833246 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:46.833250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:46.833315 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:46.833418 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:46.833932 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.834580 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.836364 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.837137 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.837322 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.837392 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:46.837663 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 02:57:46.837808 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:57:46.838075 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.832301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46067 (local address 127.19.185.194:52553)
0926 02:57:46.832924 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:46.832941 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:46.833096 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:46.833432 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:46.833437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:46.833458 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:46.833762 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:46.833771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.834715 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.834722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:46.836169 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:46.836181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.837327 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.837333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:46.837505 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.837528 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:57:46.837642 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 02:57:46.837780 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":298,"threads_started":1}
I20260926 02:57:46.838881 6961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995c3f8358c145b79903fa8c2761ccd8" candidate_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e02674f4d4240a28530920141a03a5d" is_pre_election: true
I20260926 02:57:46.839059 6700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995c3f8358c145b79903fa8c2761ccd8" candidate_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f" is_pre_election: true
I20260926 02:57:46.839530 6961 raft_consensus.cc:2454] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b13bd37d0e20418d9a3153f75e4f27a2 in term 0.
I20260926 02:57:46.839679 6700 raft_consensus.cc:2454] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b13bd37d0e20418d9a3153f75e4f27a2 in term 0.
I20260926 02:57:46.840278 6764 leader_election.cc:302] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e02674f4d4240a28530920141a03a5d, b13bd37d0e20418d9a3153f75e4f27a2; no voters:
I20260926 02:57:46.840704 7030 raft_consensus.cc:2781] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:57:46.840890 7030 raft_consensus.cc:482] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:57:46.841018 7030 raft_consensus.cc:3037] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:46.843226 7030 raft_consensus.cc:504] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38885 } } peers { permanent_uuid: "0e02674f4d4240a28530920141a03a5d" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40103 } } peers { permanent_uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46067 } }
I20260926 02:57:46.843832 7030 leader_election.cc:288] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [CANDIDATE]: Term 1 election: Requested vote from peers 0e02674f4d4240a28530920141a03a5d (127.19.185.195:40103), d1f8f0d8099f417bbc9ccd8e7aee2a7f (127.19.185.193:46067)
I20260926 02:57:46.844276 6961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995c3f8358c145b79903fa8c2761ccd8" candidate_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e02674f4d4240a28530920141a03a5d"
I20260926 02:57:46.844340 6700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995c3f8358c145b79903fa8c2761ccd8" candidate_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f"
I20260926 02:57:46.844478 6961 raft_consensus.cc:3037] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:46.844554 6700 raft_consensus.cc:3037] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:46.846701 6961 raft_consensus.cc:2454] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b13bd37d0e20418d9a3153f75e4f27a2 in term 1.
I20260926 02:57:46.846799 6700 raft_consensus.cc:2454] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b13bd37d0e20418d9a3153f75e4f27a2 in term 1.
I20260926 02:57:46.847101 6764 leader_election.cc:302] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e02674f4d4240a28530920141a03a5d, b13bd37d0e20418d9a3153f75e4f27a2; no voters:
I20260926 02:57:46.847468 7030 raft_consensus.cc:2781] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:57:46.847904 7030 raft_consensus.cc:686] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [term 1 LEADER]: Becoming Leader. State: Replica: b13bd37d0e20418d9a3153f75e4f27a2, State: Running, Role: LEADER
I20260926 02:57:46.848376 7030 consensus_queue.cc:237] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38885 } } peers { permanent_uuid: "0e02674f4d4240a28530920141a03a5d" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40103 } } peers { permanent_uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46067 } }
I20260926 02:57:46.852931 6558 catalog_manager.cc:5495] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b13bd37d0e20418d9a3153f75e4f27a2 (127.19.185.194). New cstate: current_term: 1 leader_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13bd37d0e20418d9a3153f75e4f27a2" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e02674f4d4240a28530920141a03a5d" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46067 } health_report { overall_health: UNKNOWN } } }
I20260926 02:57:46.883759 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.877947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.19.185.194:38885)
0926 02:57:46.878196 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:46.878204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:46.878324 (+ 120us) server_negotiation.cc:407] Connection header received
0926 02:57:46.878502 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:46.878506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:46.878604 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:46.878748 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:57:46.879353 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.880088 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.881595 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.882016 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.882111 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.882598 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:57:46.883080 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:57:46.883209 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:46.883430 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 02:57:46.883482 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:57:46.892119 6700 raft_consensus.cc:1261] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f [term 1 FOLLOWER]: Refusing update from remote peer b13bd37d0e20418d9a3153f75e4f27a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:57:46.892119 6961 raft_consensus.cc:1261] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d [term 1 FOLLOWER]: Refusing update from remote peer b13bd37d0e20418d9a3153f75e4f27a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:57:46.893151 7035 consensus_queue.cc:1035] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e02674f4d4240a28530920141a03a5d" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:57:46.893489 7030 consensus_queue.cc:1035] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f8f0d8099f417bbc9ccd8e7aee2a7f" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 46067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:57:46.907737 7039 mvcc.cc:205] Tried to move back new op lower bound from 7333443448379674624 to 7333443448215277568. Current Snapshot: MvccSnapshot[applied={T|T < 7333443448379674624}]
I20260926 02:57:46.911494 7038 mvcc.cc:205] Tried to move back new op lower bound from 7333443448379674624 to 7333443448215277568. Current Snapshot: MvccSnapshot[applied={T|T < 7333443448379674624 or (T in {7333443448379674624})}]
I20260926 02:57:46.920140 6558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:57:46.920441 6558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:57:46.924196 6558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e066f9151c78409da87262ecc220dfa2: Sending DeleteTablet for 3 replicas of tablet 995c3f8358c145b79903fa8c2761ccd8
I20260926 02:57:46.925418 6811 tablet_service.cc:1514] Processing DeleteTablet for tablet 995c3f8358c145b79903fa8c2761ccd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:46 UTC) from {username='slave'} at 127.0.0.1:37540
I20260926 02:57:46.925757 6680 tablet_service.cc:1514] Processing DeleteTablet for tablet 995c3f8358c145b79903fa8c2761ccd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:46 UTC) from {username='slave'} at 127.0.0.1:36948
I20260926 02:57:46.925845 6941 tablet_service.cc:1514] Processing DeleteTablet for tablet 995c3f8358c145b79903fa8c2761ccd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:46 UTC) from {username='slave'} at 127.0.0.1:55404
I20260926 02:57:46.933043 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 6618
I20260926 02:57:46.940877 7048 tablet_replica.cc:330] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2: stopping tablet replica
I20260926 02:57:46.941642 7048 raft_consensus.cc:2229] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:57:46.942538 7048 raft_consensus.cc:2258] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:46.944955 7048 ts_tablet_manager.cc:1905] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:46.952551 7050 tablet_replica.cc:330] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d: stopping tablet replica
I20260926 02:57:46.952768 7049 tablet_replica.cc:330] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f: stopping tablet replica
I20260926 02:57:46.953176 7050 raft_consensus.cc:2229] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:57:46.953202 7049 raft_consensus.cc:2229] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:57:46.953774 7050 raft_consensus.cc:2258] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:46.953797 7049 raft_consensus.cc:2258] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:46.956118 7050 ts_tablet_manager.cc:1905] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:46.956792 7048 ts_tablet_manager.cc:1918] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:46.956970 7048 log.cc:1196] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/995c3f8358c145b79903fa8c2761ccd8
I20260926 02:57:46.957573 7048 ts_tablet_manager.cc:1939] T 995c3f8358c145b79903fa8c2761ccd8 P b13bd37d0e20418d9a3153f75e4f27a2: Deleting consensus metadata
I20260926 02:57:46.959544 6543 catalog_manager.cc:4841] TS b13bd37d0e20418d9a3153f75e4f27a2 (127.19.185.194:38885): tablet 995c3f8358c145b79903fa8c2761ccd8 (table test-table [id=99996abfda5c4508baeb881a2afa5de8]) successfully deleted
I20260926 02:57:46.960012 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:46.934974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.19.185.193:46067)
0926 02:57:46.935207 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:46.935216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:46.951818 (+ 16602us) server_negotiation.cc:407] Connection header received
0926 02:57:46.952028 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:46.952036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:46.952183 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:46.952294 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:46.952984 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.954240 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.956082 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:46.956706 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:46.956819 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:46.958301 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:46.958313 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:46.958329 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:46.959071 (+ 742us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:46.959133 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:46.959158 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:46.959251 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:46.959531 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 02:57:46.959652 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:57:46.961378 6740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:46.961385 7049 ts_tablet_manager.cc:1905] T 995c3f8358c145b79903fa8c2761ccd8 P d1f8f0d8099f417bbc9ccd8e7aee2a7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:46.965312 7050 ts_tablet_manager.cc:1918] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:46.965476 7050 log.cc:1196] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/995c3f8358c145b79903fa8c2761ccd8
I20260926 02:57:46.966029 7050 ts_tablet_manager.cc:1939] T 995c3f8358c145b79903fa8c2761ccd8 P 0e02674f4d4240a28530920141a03a5d: Deleting consensus metadata
I20260926 02:57:46.967767 6543 catalog_manager.cc:4841] TS 0e02674f4d4240a28530920141a03a5d (127.19.185.195:40103): tablet 995c3f8358c145b79903fa8c2761ccd8 (table test-table [id=99996abfda5c4508baeb881a2afa5de8]) successfully deleted
I20260926 02:57:47.053603 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 6618
I20260926 02:57:47.072988 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 6749
W20260926 02:57:47.073004 6543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.185.193:46067 (error 108)
I20260926 02:57:47.075215 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:47.074680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46067 (local address 127.0.0.1:36960)
0926 02:57:47.074917 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:47.075042 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260926 02:57:47.075486 6543 catalog_manager.cc:4584] TS d1f8f0d8099f417bbc9ccd8e7aee2a7f (127.19.185.193:46067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 995c3f8358c145b79903fa8c2761ccd8: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46067: connect: Connection refused (error 111)
I20260926 02:57:47.080911 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:47.074175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.19.185.194:38885)
0926 02:57:47.074380 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:47.074392 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:47.074412 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:57:47.074478 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:47.074482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:47.074563 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:47.074693 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:47.075122 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:47.076072 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:47.077373 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:47.077844 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:47.077991 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:47.079040 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:47.079051 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:47.079070 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:47.079998 (+ 928us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:47.080050 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:47.080071 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:47.080196 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:47.080416 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 02:57:47.080514 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:57:47.081586 6871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:47.129458 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:47.128823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46067 (local address 127.0.0.1:36970)
0926 02:57:47.129090 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:47.129208 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 02:57:47.130743 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:47.130304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46067 (local address 127.0.0.1:36978)
0926 02:57:47.130481 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:47.130591 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 02:57:47.191514 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:47.190896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46067 (local address 127.0.0.1:36988)
0926 02:57:47.191150 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:47.191246 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 02:57:47.192610 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:47.192232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46067 (local address 127.0.0.1:37004)
0926 02:57:47.192430 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:47.192502 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 02:57:47.201331 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 6749
I20260926 02:57:47.220877 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 6879
I20260926 02:57:47.228282 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:47.221710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.19.185.195:40103)
0926 02:57:47.221989 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:47.221997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:47.222017 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:57:47.222224 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:47.222229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:47.222313 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:47.222446 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:47.222879 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:47.223690 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:47.225006 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:47.225394 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:47.225540 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:47.226683 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:47.226694 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:47.226710 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:47.227399 (+ 689us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:47.227453 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:47.227476 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:47.227601 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:47.227873 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 02:57:47.227989 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:57:47.229076 7001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:47.274030 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:47.273460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46067 (local address 127.0.0.1:37020)
0926 02:57:47.273704 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:47.273822 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 02:57:47.275338 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:47.274819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46067 (local address 127.0.0.1:37034)
0926 02:57:47.275065 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:47.275162 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260926 02:57:47.319519 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 6879
I20260926 02:57:47.336362 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 6528
I20260926 02:57:47.337693 6589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:47.451911 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:47.451053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:46067 (local address 127.0.0.1:37048)
0926 02:57:47.451625 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:47.451736 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.193:46067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402}
I20260926 02:57:47.452050 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 6528
2026-09-26T02:57:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2827 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:57:47 dist-test-slave-ktng krb5kdc[7059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:57:47 dist-test-slave-ktng krb5kdc[7059](info): set up 2 sockets
Sep 26 02:57:47 dist-test-slave-ktng krb5kdc[7059](info): commencing operation
krb5kdc: starting...
W20260926 02:57:48.580495 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:57:48 dist-test-slave-ktng krb5kdc[7059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:57:48.843935 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:34369
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:34369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:49.183398 7072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:49.183750 7072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:49.183902 7072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:49.191250 7072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:57:49.191454 7072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:49.191581 7072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:49.191679 7072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:57:49.191763 7072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:57:49.201876 7072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:34369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:34369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:49.203382 7072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:49.204770 7072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:49.217222 7078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:49.217196 7077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:49.217448 7080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:49.218421 7072 server_base.cc:1034] running on GCE node
I20260926 02:57:49.219220 7072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:49.220674 7072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:49.221926 7072 hybrid_clock.cc:648] HybridClock initialized: now 1790391469221842 us; error 73 us; skew 500 ppm
I20260926 02:57:49.222301 7072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:49 dist-test-slave-ktng krb5kdc[7059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391469, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:49.225533 7072 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:57:49.227006 7072 webserver.cc:460] Webserver started at http://127.19.185.254:39213/ using document root <none> and password file <none>
I20260926 02:57:49.227814 7072 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:49.227965 7072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:49.228300 7072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:49.231361 7072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "3e3c41cd3a7f47cebb90a9aa6ab98d8b"
format_stamp: "Formatted at 2026-09-26 02:57:49 on dist-test-slave-ktng"
server_key: "b457cd0e89714e7e00174f57a5bfb0c2"
server_key_iv: "78985653068f31966264a08457e55edc"
server_key_version: "encryptionkey@0"
I20260926 02:57:49.232125 7072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "3e3c41cd3a7f47cebb90a9aa6ab98d8b"
format_stamp: "Formatted at 2026-09-26 02:57:49 on dist-test-slave-ktng"
server_key: "b457cd0e89714e7e00174f57a5bfb0c2"
server_key_iv: "78985653068f31966264a08457e55edc"
server_key_version: "encryptionkey@0"
I20260926 02:57:49.237325 7072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:57:49.240794 7087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:49.242138 7072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:57:49.242326 7072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "3e3c41cd3a7f47cebb90a9aa6ab98d8b"
format_stamp: "Formatted at 2026-09-26 02:57:49 on dist-test-slave-ktng"
server_key: "b457cd0e89714e7e00174f57a5bfb0c2"
server_key_iv: "78985653068f31966264a08457e55edc"
server_key_version: "encryptionkey@0"
I20260926 02:57:49.242537 7072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:49.257505 7072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:49.261525 7072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:49.261824 7072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:49.280656 7072 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:34369
I20260926 02:57:49.280684 7138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:34369 every 8 connection(s)
I20260926 02:57:49.282127 7072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:57:49.285962 7139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:49.288905 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 7072
I20260926 02:57:49.289089 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:57:49.289624 20199 external_mini_cluster.cc:1428] Setting key 9e7de724a35b64542a3d657d8f959ae8
I20260926 02:57:49.298763 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b: Bootstrap starting.
I20260926 02:57:49.302680 7139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:49.304116 7139 log.cc:824] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:57:49 dist-test-slave-ktng krb5kdc[7059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:57:49.307456 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b: No bootstrap required, opened a new log
I20260926 02:57:49.313874 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:49.291597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.19.185.254:34369)
0926 02:57:49.292349 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:49.292378 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:49.292424 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:57:49.294145 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:49.294169 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:49.294600 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:49.294946 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:49.296815 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:49.298116 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:49.299906 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:49.300605 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:49.300735 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:49.306021 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:49.306028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:49.306043 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:49.306082 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:49.309557 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:49.310320 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:49.310327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:49.310333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:49.310508 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:49.310938 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:49.310941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:49.310944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:49.311555 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:49.311960 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:49.312410 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 02:57:49.312842 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":328,"threads_started":1}
I20260926 02:57:49.316051 7139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3c41cd3a7f47cebb90a9aa6ab98d8b" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34369 } }
I20260926 02:57:49.316455 7139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:49.316587 7139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e3c41cd3a7f47cebb90a9aa6ab98d8b, State: Initialized, Role: FOLLOWER
I20260926 02:57:49.317268 7139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3c41cd3a7f47cebb90a9aa6ab98d8b" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34369 } }
I20260926 02:57:49.317520 7139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:57:49.317672 7139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:57:49.317878 7139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:49.320150 7139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3c41cd3a7f47cebb90a9aa6ab98d8b" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34369 } }
I20260926 02:57:49.320662 7139 leader_election.cc:302] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e3c41cd3a7f47cebb90a9aa6ab98d8b; no voters:
I20260926 02:57:49.321367 7139 leader_election.cc:288] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:57:49.321502 7144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:57:49.322279 7144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [term 1 LEADER]: Becoming Leader. State: Replica: 3e3c41cd3a7f47cebb90a9aa6ab98d8b, State: Running, Role: LEADER
I20260926 02:57:49.322857 7144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3c41cd3a7f47cebb90a9aa6ab98d8b" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34369 } }
I20260926 02:57:49.323720 7139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:57:49.326186 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e3c41cd3a7f47cebb90a9aa6ab98d8b. Latest consensus state: current_term: 1 leader_uuid: "3e3c41cd3a7f47cebb90a9aa6ab98d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3c41cd3a7f47cebb90a9aa6ab98d8b" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34369 } } }
I20260926 02:57:49.327194 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:49.328258 7145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e3c41cd3a7f47cebb90a9aa6ab98d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3c41cd3a7f47cebb90a9aa6ab98d8b" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 34369 } } }
I20260926 02:57:49.328802 7145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:49.329537 7152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:57:49.334800 7152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:57:49.343863 7152 catalog_manager.cc:1347] Generated new cluster ID: d57d26af45e94d1c9a0a6f3716e8dd26
I20260926 02:57:49.344000 7152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:57:49.375653 7152 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:57:49.377034 7152 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:57:49.386543 7152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b: Generated new TSK 0
I20260926 02:57:49.387488 7152 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.185.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
Entry for principal kudu/127.19.185.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab.
I20260926 02:57:49.454975 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34369
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:49.805195 7165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:49.805502 7165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:49.805693 7165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:49.813396 7165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:49.813546 7165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:49.813725 7165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:57:49.824227 7165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.193.keytab
--principal=kudu/127.19.185.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:49.825743 7165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:49.827142 7165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:49.841053 7170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:49.841106 7173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:49.841214 7165 server_base.cc:1034] running on GCE node
W20260926 02:57:49.841106 7171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:49.842180 7165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:49.843106 7165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:49.844355 7165 hybrid_clock.cc:648] HybridClock initialized: now 1790391469844275 us; error 60 us; skew 500 ppm
I20260926 02:57:49.844707 7165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:49 dist-test-slave-ktng krb5kdc[7059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391469, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:49.848091 7165 init.cc:377] Logged in from keytab as kudu/127.19.185.193@KRBTEST.COM (short username kudu)
I20260926 02:57:49.849562 7165 webserver.cc:460] Webserver started at http://127.19.185.193:37883/ using document root <none> and password file <none>
I20260926 02:57:49.850255 7165 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:49.850409 7165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:49.850731 7165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:49.853749 7165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "f5b32ec0f42c452c940ce2fd9e50c720"
format_stamp: "Formatted at 2026-09-26 02:57:49 on dist-test-slave-ktng"
server_key: "e9248bf4f20709a71708f36cf4659d81"
server_key_iv: "42de7fbeba2b91f29b701be69f4f2c55"
server_key_version: "encryptionkey@0"
I20260926 02:57:49.854533 7165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "f5b32ec0f42c452c940ce2fd9e50c720"
format_stamp: "Formatted at 2026-09-26 02:57:49 on dist-test-slave-ktng"
server_key: "e9248bf4f20709a71708f36cf4659d81"
server_key_iv: "42de7fbeba2b91f29b701be69f4f2c55"
server_key_version: "encryptionkey@0"
I20260926 02:57:49.859861 7165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 02:57:49.863415 7180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:49.864850 7165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:57:49.865028 7165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "f5b32ec0f42c452c940ce2fd9e50c720"
format_stamp: "Formatted at 2026-09-26 02:57:49 on dist-test-slave-ktng"
server_key: "e9248bf4f20709a71708f36cf4659d81"
server_key_iv: "42de7fbeba2b91f29b701be69f4f2c55"
server_key_version: "encryptionkey@0"
I20260926 02:57:49.865236 7165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:49.896723 7165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:49.901062 7165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:49.901365 7165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:49.902377 7165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:49.903844 7165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:49.903954 7165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:49.904103 7165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:49.904183 7165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:49.938009 7165 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:35203
I20260926 02:57:49.938098 7292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:35203 every 8 connection(s)
I20260926 02:57:49.939380 7165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:57:49.943717 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 7165
I20260926 02:57:49.943915 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:57:49.944383 20199 external_mini_cluster.cc:1428] Setting key c30ea1ded82d238d3d22d946de4fb7ab
Sep 26 02:57:49 dist-test-slave-ktng krb5kdc[7059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391469, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.193@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:57:49.961524 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:49.942260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:35949 (local address 127.19.185.254:34369)
0926 02:57:49.942788 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:49.942793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:49.943718 (+ 925us) server_negotiation.cc:407] Connection header received
0926 02:57:49.946047 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:49.946053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:49.946142 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:49.946278 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:49.948160 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:49.948905 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:49.950479 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:49.950885 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:49.951023 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:49.955272 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:49.955276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:49.955279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:49.955387 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:49.957960 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:49.959014 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:49.959018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:49.959020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:49.959135 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:49.959680 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:49.959684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:49.959686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:49.959960 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:49.960070 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:49.961070 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0926 02:57:49.961237 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":273,"threads_started":1}
I20260926 02:57:49.963384 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:49.942755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:34369 (local address 127.19.185.193:35949)
0926 02:57:49.943551 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:49.943596 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:49.945724 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:49.946526 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:49.946534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:49.947336 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:49.947985 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:49.948006 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:49.949081 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:49.949085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:49.950330 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:49.950342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:49.951123 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:49.951131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:49.951441 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:49.952198 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:57:49.952224 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:49.954994 (+ 2770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:49.958110 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:49.958122 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:49.958145 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:49.958866 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:49.959251 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:49.959259 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:49.959266 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:49.959561 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:49.960093 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:49.960105 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:49.960616 (+ 511us) client_negotiation.cc:769] Sending connection context
0926 02:57:49.960961 (+ 345us) client_negotiation.cc:240] Negotiation successful
0926 02:57:49.961594 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":307,"threads_started":1}
I20260926 02:57:49.965133 7293 heartbeater.cc:347] Connected to a master server at 127.19.185.254:34369
I20260926 02:57:49.965447 7293 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:49.966184 7293 heartbeater.cc:511] Master 127.19.185.254:34369 requested a full tablet report, sending...
I20260926 02:57:49.968474 7104 ts_manager.cc:194] Registered new tserver with Master: f5b32ec0f42c452c940ce2fd9e50c720 (127.19.185.193:35203)
I20260926 02:57:49.970181 7104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.193@KRBTEST.COM'} at 127.19.185.193:35949
WARNING: no policy specified for kudu/127.19.185.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
Entry for principal kudu/127.19.185.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab.
I20260926 02:57:50.007180 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34369
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:50.337273 7299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:50.337657 7299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:50.337857 7299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:50.345212 7299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:50.345368 7299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:50.345556 7299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:57:50.355552 7299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.194.keytab
--principal=kudu/127.19.185.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:50.356949 7299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:50.358317 7299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:50.372563 7304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:50.372517 7305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:50.372617 7307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:50.376899 7299 server_base.cc:1034] running on GCE node
I20260926 02:57:50.377503 7299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:50.378487 7299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:50.379701 7299 hybrid_clock.cc:648] HybridClock initialized: now 1790391470379644 us; error 57 us; skew 500 ppm
I20260926 02:57:50.380028 7299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:50 dist-test-slave-ktng krb5kdc[7059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391470, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:50.383064 7299 init.cc:377] Logged in from keytab as kudu/127.19.185.194@KRBTEST.COM (short username kudu)
I20260926 02:57:50.384660 7299 webserver.cc:460] Webserver started at http://127.19.185.194:40369/ using document root <none> and password file <none>
I20260926 02:57:50.385305 7299 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:50.385404 7299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:50.385668 7299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:50.388617 7299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "21f61263791f45c5a9c9594389443cde"
format_stamp: "Formatted at 2026-09-26 02:57:50 on dist-test-slave-ktng"
server_key: "6c18a24445ddccf3be83c9facf98dba9"
server_key_iv: "8b903c19d752e5a8fd59f42f04e63cf5"
server_key_version: "encryptionkey@0"
I20260926 02:57:50.389390 7299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "21f61263791f45c5a9c9594389443cde"
format_stamp: "Formatted at 2026-09-26 02:57:50 on dist-test-slave-ktng"
server_key: "6c18a24445ddccf3be83c9facf98dba9"
server_key_iv: "8b903c19d752e5a8fd59f42f04e63cf5"
server_key_version: "encryptionkey@0"
I20260926 02:57:50.394490 7299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:57:50.397943 7314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:50.399463 7299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:57:50.399593 7299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "21f61263791f45c5a9c9594389443cde"
format_stamp: "Formatted at 2026-09-26 02:57:50 on dist-test-slave-ktng"
server_key: "6c18a24445ddccf3be83c9facf98dba9"
server_key_iv: "8b903c19d752e5a8fd59f42f04e63cf5"
server_key_version: "encryptionkey@0"
I20260926 02:57:50.399771 7299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:50.416575 7299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:50.420944 7299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:50.421202 7299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:50.422212 7299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:50.423543 7299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:50.423614 7299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:50.423698 7299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:50.423741 7299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:50.458366 7299 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:46001
I20260926 02:57:50.458542 7426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:46001 every 8 connection(s)
I20260926 02:57:50.459767 7299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:57:50.463093 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 7299
I20260926 02:57:50.463366 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:57:50.463968 20199 external_mini_cluster.cc:1428] Setting key 4632886e6ff7e6d994a9e3d0e5b2f183
Sep 26 02:57:50 dist-test-slave-ktng krb5kdc[7059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391470, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.194@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:57:50.483103 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:50.462354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:60443 (local address 127.19.185.254:34369)
0926 02:57:50.462803 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:50.462808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:50.463934 (+ 1126us) server_negotiation.cc:407] Connection header received
0926 02:57:50.466335 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:50.466340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:50.466438 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:50.466555 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:50.468086 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:50.468863 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:50.471468 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:50.471894 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:50.471992 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:50.476426 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:50.476431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:50.476434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:50.476466 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:50.479784 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:50.480704 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:50.480707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:50.480709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:50.480847 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:50.481357 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:50.481360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:50.481362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:50.481652 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:50.481763 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:50.482660 (+ 897us) server_negotiation.cc:299] Negotiation successful
0926 02:57:50.482825 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":244,"threads_started":1}
I20260926 02:57:50.484671 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:50.462925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:34369 (local address 127.19.185.194:60443)
0926 02:57:50.463780 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:50.463816 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:50.466077 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:50.466796 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:50.466805 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:50.467347 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:50.467921 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:50.467938 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:50.469067 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:50.469075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:50.471265 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:50.471283 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:50.472125 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:50.472134 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:50.472468 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:50.473200 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:57:50.473232 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:50.476164 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:50.479931 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:50.479943 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:50.479962 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:50.480575 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:50.480972 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:50.480980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:50.480987 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:50.481242 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:50.481778 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:50.481799 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:50.482262 (+ 463us) client_negotiation.cc:769] Sending connection context
0926 02:57:50.482578 (+ 316us) client_negotiation.cc:240] Negotiation successful
0926 02:57:50.483159 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":389,"threads_started":1}
I20260926 02:57:50.486282 7427 heartbeater.cc:347] Connected to a master server at 127.19.185.254:34369
I20260926 02:57:50.486601 7427 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:50.487525 7427 heartbeater.cc:511] Master 127.19.185.254:34369 requested a full tablet report, sending...
I20260926 02:57:50.489149 7104 ts_manager.cc:194] Registered new tserver with Master: 21f61263791f45c5a9c9594389443cde (127.19.185.194:46001)
I20260926 02:57:50.490048 7104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.194@KRBTEST.COM'} at 127.19.185.194:60443
WARNING: no policy specified for kudu/127.19.185.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
Entry for principal kudu/127.19.185.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab.
I20260926 02:57:50.527000 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34369
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:50.877887 7433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:50.878181 7433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:50.878281 7433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:50.885758 7433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:50.885872 7433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:50.885985 7433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:57:50.896034 7433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.195.keytab
--principal=kudu/127.19.185.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:34369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:50.897361 7433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:50.898644 7433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:50.912293 7438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:50.912390 7441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:50.912319 7439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:50.913949 7433 server_base.cc:1034] running on GCE node
I20260926 02:57:50.914644 7433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:50.915830 7433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:50.917093 7433 hybrid_clock.cc:648] HybridClock initialized: now 1790391470917054 us; error 59 us; skew 500 ppm
I20260926 02:57:50.917455 7433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:50 dist-test-slave-ktng krb5kdc[7059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391470, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:50.927336 7433 init.cc:377] Logged in from keytab as kudu/127.19.185.195@KRBTEST.COM (short username kudu)
I20260926 02:57:50.929184 7433 webserver.cc:460] Webserver started at http://127.19.185.195:37151/ using document root <none> and password file <none>
I20260926 02:57:50.930105 7433 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:50.930285 7433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:50.930697 7433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:50.934041 7433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "436198a742e64497bab127c016ff1002"
format_stamp: "Formatted at 2026-09-26 02:57:50 on dist-test-slave-ktng"
server_key: "0f0faa47e1006a0f29bdb570c171be09"
server_key_iv: "864a0cb2271bb9575bc294bd9f87eeee"
server_key_version: "encryptionkey@0"
I20260926 02:57:50.934897 7433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "436198a742e64497bab127c016ff1002"
format_stamp: "Formatted at 2026-09-26 02:57:50 on dist-test-slave-ktng"
server_key: "0f0faa47e1006a0f29bdb570c171be09"
server_key_iv: "864a0cb2271bb9575bc294bd9f87eeee"
server_key_version: "encryptionkey@0"
I20260926 02:57:50.940932 7433 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260926 02:57:50.945034 7448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:50.946623 7433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:57:50.946823 7433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "436198a742e64497bab127c016ff1002"
format_stamp: "Formatted at 2026-09-26 02:57:50 on dist-test-slave-ktng"
server_key: "0f0faa47e1006a0f29bdb570c171be09"
server_key_iv: "864a0cb2271bb9575bc294bd9f87eeee"
server_key_version: "encryptionkey@0"
I20260926 02:57:50.947170 7433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:50.970881 7433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:50.973407 7293 heartbeater.cc:503] Master 127.19.185.254:34369 was elected leader, sending a full tablet report...
I20260926 02:57:50.976316 7433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:50.976671 7433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:50.977777 7433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:50.979349 7433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:50.979470 7433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 02:57:50.979640 7433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:50.979722 7433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:51.016551 7433 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:34127
I20260926 02:57:51.016597 7560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:34127 every 8 connection(s)
I20260926 02:57:51.018165 7433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:57:51.023527 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 7433
I20260926 02:57:51.023738 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:57:51.024235 20199 external_mini_cluster.cc:1428] Setting key 2525806dcb2a402503979f5aeb5b9423
Sep 26 02:57:51 dist-test-slave-ktng krb5kdc[7059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391470, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.195@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:57:51.041042 7563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.021010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:34333 (local address 127.19.185.254:34369)
0926 02:57:51.021484 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:51.021488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:51.022482 (+ 994us) server_negotiation.cc:407] Connection header received
0926 02:57:51.024738 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:51.024744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:51.024840 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:51.024988 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:51.026759 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.027841 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.029543 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.030045 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.030211 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.034232 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:51.034237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:51.034241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:51.034283 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:51.037556 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:51.038461 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:51.038469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:51.038472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:51.038616 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:51.039182 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:51.039188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:51.039191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:51.039623 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:51.039790 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:51.040546 (+ 756us) server_negotiation.cc:299] Negotiation successful
0926 02:57:51.040729 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":256,"threads_started":1}
I20260926 02:57:51.041994 7564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.021568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:34369 (local address 127.19.185.195:34333)
0926 02:57:51.022287 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:51.022333 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:51.024435 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:51.025256 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:51.025269 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:51.025877 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:51.026578 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:51.026604 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.028008 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.028012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:51.029395 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:51.029408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.030323 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.030328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:51.030548 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.031365 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:57:51.031416 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:51.034004 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:51.037733 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:51.037743 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:51.037761 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:51.038321 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:51.038738 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:57:51.038743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:57:51.038748 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:57:51.039035 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:57:51.039759 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:51.039765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:51.040165 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 02:57:51.040456 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 02:57:51.040857 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":281,"threads_started":1}
I20260926 02:57:51.043567 7561 heartbeater.cc:347] Connected to a master server at 127.19.185.254:34369
I20260926 02:57:51.043874 7561 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:51.044611 7561 heartbeater.cc:511] Master 127.19.185.254:34369 requested a full tablet report, sending...
I20260926 02:57:51.046125 7104 ts_manager.cc:194] Registered new tserver with Master: 436198a742e64497bab127c016ff1002 (127.19.185.195:34127)
I20260926 02:57:51.046998 7104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.185.195@KRBTEST.COM'} at 127.19.185.195:34333
I20260926 02:57:51.052738 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:57:51.070648 7563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.056992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.19.185.254:34369)
0926 02:57:51.057209 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:51.057218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:51.057672 (+ 454us) server_negotiation.cc:407] Connection header received
0926 02:57:51.057821 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:51.057825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:51.057951 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:51.058041 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:51.059536 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.060300 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.061583 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.062035 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.062138 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.064614 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:51.064620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:51.064623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:51.064657 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:51.067672 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:51.068357 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:51.068362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:51.068365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:51.068669 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:51.069020 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:51.069024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:51.069026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:51.069371 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:51.069568 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:51.070075 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 02:57:51.070267 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:57:51.074798 7104 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:57:51.081771 7104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:57:51.117928 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.094692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.19.185.194:46001)
0926 02:57:51.105976 (+ 11284us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:51.105987 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:51.108341 (+ 2354us) server_negotiation.cc:407] Connection header received
0926 02:57:51.108448 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:51.108455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:51.108714 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:51.108948 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:51.110445 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.112058 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.114420 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.116033 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.116193 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.116382 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:51.117330 (+ 948us) server_negotiation.cc:299] Negotiation successful
0926 02:57:51.117528 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11094,"thread_start_us":10935,"threads_started":1}
I20260926 02:57:51.117928 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.094699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.19.185.193:35203)
0926 02:57:51.105421 (+ 10722us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:51.105438 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:51.108922 (+ 3484us) server_negotiation.cc:407] Connection header received
0926 02:57:51.109030 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:51.109037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:51.109295 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:51.109479 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:51.110325 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.112020 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.115034 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.116656 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.116882 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.117079 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:51.117372 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 02:57:51.117537 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10460,"thread_start_us":10259,"threads_started":1}
I20260926 02:57:51.118014 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.094403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:35203 (local address 127.0.0.1:40820)
0926 02:57:51.108635 (+ 14232us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:51.108651 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:51.108769 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:51.109709 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:51.109731 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:51.109752 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:51.110135 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:51.110144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.112208 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.112213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:51.114871 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:51.114882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.116885 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.116892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:51.117095 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.117137 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 02:57:51.117269 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 02:57:51.117441 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14109,"mutex_wait_us":187,"thread_start_us":14770,"threads_started":1}
I20260926 02:57:51.118036 7577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.094154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:46001 (local address 127.0.0.1:41518)
0926 02:57:51.107989 (+ 13835us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:51.108013 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:51.108171 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:51.109472 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:51.109477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:51.109538 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:51.110191 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:51.110208 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.112208 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.112213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:51.114255 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:51.114269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.116558 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.116566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:51.116974 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.117091 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 02:57:51.117228 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 02:57:51.117441 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13672,"mutex_wait_us":256,"thread_start_us":14220,"threads_started":1}
I20260926 02:57:51.119071 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.094039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.19.185.195:34127)
0926 02:57:51.108247 (+ 14208us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:51.108259 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:51.108300 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:57:51.108422 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:51.108431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:51.108714 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:51.109727 (+ 1013us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:51.110049 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.111343 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.114660 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.116033 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.116220 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.116396 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:51.118444 (+ 2048us) server_negotiation.cc:299] Negotiation successful
0926 02:57:51.118632 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14020,"thread_start_us":13888,"threads_started":1}
I20260926 02:57:51.119086 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.093706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:34127 (local address 127.0.0.1:39152)
0926 02:57:51.106499 (+ 12793us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:51.106563 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:51.106838 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:51.109002 (+ 2164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:51.109013 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:51.109053 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:51.109489 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:51.109502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.111500 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.111508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:51.114464 (+ 2956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:51.114482 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.116576 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.116581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:51.116970 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.117091 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 02:57:51.117228 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 02:57:51.118305 (+ 1077us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12409,"mutex_wait_us":987,"thread_start_us":13186,"threads_started":1}
I20260926 02:57:51.122403 7228 tablet_service.cc:1467] Processing CreateTablet for tablet 64480cde48d6491686b788b73d64e697 (DEFAULT_TABLE table=test-table [id=7037d10dc5764cb286c47d7e0b4b6114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:51.122850 7362 tablet_service.cc:1467] Processing CreateTablet for tablet 64480cde48d6491686b788b73d64e697 (DEFAULT_TABLE table=test-table [id=7037d10dc5764cb286c47d7e0b4b6114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:51.122784 7496 tablet_service.cc:1467] Processing CreateTablet for tablet 64480cde48d6491686b788b73d64e697 (DEFAULT_TABLE table=test-table [id=7037d10dc5764cb286c47d7e0b4b6114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:51.123977 7228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64480cde48d6491686b788b73d64e697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:51.124462 7496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64480cde48d6491686b788b73d64e697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:51.124954 7362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64480cde48d6491686b788b73d64e697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:51.136691 7580 tablet_bootstrap.cc:492] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720: Bootstrap starting.
I20260926 02:57:51.140234 7580 tablet_bootstrap.cc:654] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:51.141247 7581 tablet_bootstrap.cc:492] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002: Bootstrap starting.
I20260926 02:57:51.141734 7580 log.cc:824] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:51.145617 7581 tablet_bootstrap.cc:654] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:51.147344 7581 log.cc:824] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:51.147612 7582 tablet_bootstrap.cc:492] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde: Bootstrap starting.
I20260926 02:57:51.150977 7581 tablet_bootstrap.cc:492] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002: No bootstrap required, opened a new log
I20260926 02:57:51.151434 7582 tablet_bootstrap.cc:654] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:51.151474 7581 ts_tablet_manager.cc:1397] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20260926 02:57:51.152407 7580 tablet_bootstrap.cc:492] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720: No bootstrap required, opened a new log
I20260926 02:57:51.152680 7580 ts_tablet_manager.cc:1397] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.012s
I20260926 02:57:51.152940 7582 log.cc:824] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:51.156173 7582 tablet_bootstrap.cc:492] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde: No bootstrap required, opened a new log
I20260926 02:57:51.156461 7582 ts_tablet_manager.cc:1397] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260926 02:57:51.162978 7581 raft_consensus.cc:348] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436198a742e64497bab127c016ff1002" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 34127 } } peers { permanent_uuid: "21f61263791f45c5a9c9594389443cde" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 46001 } } peers { permanent_uuid: "f5b32ec0f42c452c940ce2fd9e50c720" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35203 } }
I20260926 02:57:51.163496 7581 raft_consensus.cc:374] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:51.163244 7580 raft_consensus.cc:348] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436198a742e64497bab127c016ff1002" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 34127 } } peers { permanent_uuid: "21f61263791f45c5a9c9594389443cde" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 46001 } } peers { permanent_uuid: "f5b32ec0f42c452c940ce2fd9e50c720" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35203 } }
I20260926 02:57:51.163589 7581 raft_consensus.cc:729] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436198a742e64497bab127c016ff1002, State: Initialized, Role: FOLLOWER
I20260926 02:57:51.163682 7580 raft_consensus.cc:374] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:51.163762 7580 raft_consensus.cc:729] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b32ec0f42c452c940ce2fd9e50c720, State: Initialized, Role: FOLLOWER
I20260926 02:57:51.164350 7581 consensus_queue.cc:260] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436198a742e64497bab127c016ff1002" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 34127 } } peers { permanent_uuid: "21f61263791f45c5a9c9594389443cde" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 46001 } } peers { permanent_uuid: "f5b32ec0f42c452c940ce2fd9e50c720" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35203 } }
I20260926 02:57:51.164351 7580 consensus_queue.cc:260] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436198a742e64497bab127c016ff1002" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 34127 } } peers { permanent_uuid: "21f61263791f45c5a9c9594389443cde" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 46001 } } peers { permanent_uuid: "f5b32ec0f42c452c940ce2fd9e50c720" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35203 } }
I20260926 02:57:51.169392 7582 raft_consensus.cc:348] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436198a742e64497bab127c016ff1002" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 34127 } } peers { permanent_uuid: "21f61263791f45c5a9c9594389443cde" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 46001 } } peers { permanent_uuid: "f5b32ec0f42c452c940ce2fd9e50c720" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35203 } }
I20260926 02:57:51.170038 7582 raft_consensus.cc:374] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:51.170198 7582 raft_consensus.cc:729] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f61263791f45c5a9c9594389443cde, State: Initialized, Role: FOLLOWER
I20260926 02:57:51.170953 7582 consensus_queue.cc:260] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436198a742e64497bab127c016ff1002" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 34127 } } peers { permanent_uuid: "21f61263791f45c5a9c9594389443cde" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 46001 } } peers { permanent_uuid: "f5b32ec0f42c452c940ce2fd9e50c720" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35203 } }
I20260926 02:57:51.172272 7427 heartbeater.cc:503] Master 127.19.185.254:34369 was elected leader, sending a full tablet report...
I20260926 02:57:51.173023 7582 ts_tablet_manager.cc:1428] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260926 02:57:51.173668 7561 heartbeater.cc:503] Master 127.19.185.254:34369 was elected leader, sending a full tablet report...
I20260926 02:57:51.174000 7580 ts_tablet_manager.cc:1428] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260926 02:57:51.174432 7581 ts_tablet_manager.cc:1428] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002: Time spent starting tablet: real 0.023s user 0.020s sys 0.000s
W20260926 02:57:51.191933 7294 tablet.cc:2369] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:57:51.211834 7428 tablet.cc:2369] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:57:51.270085 7562 tablet.cc:2369] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:57:51.460085 7586 raft_consensus.cc:482] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:57:51.460421 7586 raft_consensus.cc:504] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436198a742e64497bab127c016ff1002" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 34127 } } peers { permanent_uuid: "21f61263791f45c5a9c9594389443cde" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 46001 } } peers { permanent_uuid: "f5b32ec0f42c452c940ce2fd9e50c720" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35203 } }
I20260926 02:57:51.462232 7586 leader_election.cc:288] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21f61263791f45c5a9c9594389443cde (127.19.185.194:46001), f5b32ec0f42c452c940ce2fd9e50c720 (127.19.185.193:35203)
I20260926 02:57:51.469377 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.463181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:38405 (local address 127.19.185.193:35203)
0926 02:57:51.463411 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:51.463420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:51.463574 (+ 154us) server_negotiation.cc:407] Connection header received
0926 02:57:51.463831 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:51.463838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:51.463928 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:51.464019 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:51.464543 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.465414 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.467104 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.468386 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.468594 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.468677 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:51.468986 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 02:57:51.469129 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:57:51.469691 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.462688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:51917 (local address 127.19.185.194:46001)
0926 02:57:51.463118 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:51.463124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:51.463147 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:57:51.463224 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:51.463231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:51.463355 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:51.463449 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:51.463865 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.464891 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.467395 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.468799 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.468972 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.469047 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:51.469306 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 02:57:51.469436 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287}
I20260926 02:57:51.470099 7564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.462471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:46001 (local address 127.19.185.195:51917)
0926 02:57:51.462834 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:51.462856 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:51.462984 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:51.463491 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:51.463496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:51.463527 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:51.463717 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:51.463727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.465035 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.465040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:51.467178 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:51.467193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.468958 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.468964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:51.469153 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.469177 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:57:51.469280 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 02:57:51.469424 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":69}
I20260926 02:57:51.470125 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.462656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:35203 (local address 127.19.185.195:38405)
0926 02:57:51.463484 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:51.463503 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:51.463684 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:51.464050 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:51.464055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:51.464077 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:51.464385 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:51.464394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.465581 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.465585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:51.466956 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:51.466968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.468600 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.468607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:51.468786 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.468821 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:57:51.469659 (+ 838us) client_negotiation.cc:240] Negotiation successful
0926 02:57:51.469793 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":526,"threads_started":1}
I20260926 02:57:51.471266 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64480cde48d6491686b788b73d64e697" candidate_uuid: "436198a742e64497bab127c016ff1002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f61263791f45c5a9c9594389443cde" is_pre_election: true
I20260926 02:57:51.471963 7382 raft_consensus.cc:2454] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 436198a742e64497bab127c016ff1002 in term 0.
I20260926 02:57:51.472051 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64480cde48d6491686b788b73d64e697" candidate_uuid: "436198a742e64497bab127c016ff1002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b32ec0f42c452c940ce2fd9e50c720" is_pre_election: true
I20260926 02:57:51.472563 7248 raft_consensus.cc:2454] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 436198a742e64497bab127c016ff1002 in term 0.
I20260926 02:57:51.472766 7449 leader_election.cc:302] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f61263791f45c5a9c9594389443cde, 436198a742e64497bab127c016ff1002; no voters:
I20260926 02:57:51.473184 7586 raft_consensus.cc:2781] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:57:51.473322 7586 raft_consensus.cc:482] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:57:51.473424 7586 raft_consensus.cc:3037] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:51.475780 7586 raft_consensus.cc:504] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436198a742e64497bab127c016ff1002" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 34127 } } peers { permanent_uuid: "21f61263791f45c5a9c9594389443cde" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 46001 } } peers { permanent_uuid: "f5b32ec0f42c452c940ce2fd9e50c720" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35203 } }
I20260926 02:57:51.476378 7586 leader_election.cc:288] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [CANDIDATE]: Term 1 election: Requested vote from peers 21f61263791f45c5a9c9594389443cde (127.19.185.194:46001), f5b32ec0f42c452c940ce2fd9e50c720 (127.19.185.193:35203)
I20260926 02:57:51.476794 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64480cde48d6491686b788b73d64e697" candidate_uuid: "436198a742e64497bab127c016ff1002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f61263791f45c5a9c9594389443cde"
I20260926 02:57:51.477000 7382 raft_consensus.cc:3037] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:51.476919 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64480cde48d6491686b788b73d64e697" candidate_uuid: "436198a742e64497bab127c016ff1002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b32ec0f42c452c940ce2fd9e50c720"
I20260926 02:57:51.477154 7248 raft_consensus.cc:3037] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:51.479182 7382 raft_consensus.cc:2454] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436198a742e64497bab127c016ff1002 in term 1.
I20260926 02:57:51.479410 7248 raft_consensus.cc:2454] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436198a742e64497bab127c016ff1002 in term 1.
I20260926 02:57:51.479597 7449 leader_election.cc:302] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f61263791f45c5a9c9594389443cde, 436198a742e64497bab127c016ff1002; no voters:
I20260926 02:57:51.479941 7586 raft_consensus.cc:2781] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:57:51.480405 7586 raft_consensus.cc:686] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [term 1 LEADER]: Becoming Leader. State: Replica: 436198a742e64497bab127c016ff1002, State: Running, Role: LEADER
I20260926 02:57:51.480885 7586 consensus_queue.cc:237] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436198a742e64497bab127c016ff1002" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 34127 } } peers { permanent_uuid: "21f61263791f45c5a9c9594389443cde" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 46001 } } peers { permanent_uuid: "f5b32ec0f42c452c940ce2fd9e50c720" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35203 } }
I20260926 02:57:51.485814 7104 catalog_manager.cc:5495] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 reported cstate change: term changed from 0 to 1, leader changed from <none> to 436198a742e64497bab127c016ff1002 (127.19.185.195). New cstate: current_term: 1 leader_uuid: "436198a742e64497bab127c016ff1002" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436198a742e64497bab127c016ff1002" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 34127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21f61263791f45c5a9c9594389443cde" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 46001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5b32ec0f42c452c940ce2fd9e50c720" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35203 } health_report { overall_health: UNKNOWN } } }
I20260926 02:57:51.582614 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.577252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.19.185.195:34127)
0926 02:57:51.577465 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:51.577469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:51.577793 (+ 324us) server_negotiation.cc:407] Connection header received
0926 02:57:51.577976 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:51.577980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:51.578064 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:51.578189 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:57:51.578574 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.579269 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.580654 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.581036 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.581129 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.581557 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:57:51.582068 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:57:51.582179 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:51.582327 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 02:57:51.582392 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:57:51.590715 7382 raft_consensus.cc:1261] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde [term 1 FOLLOWER]: Refusing update from remote peer 436198a742e64497bab127c016ff1002: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:57:51.590708 7248 raft_consensus.cc:1261] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720 [term 1 FOLLOWER]: Refusing update from remote peer 436198a742e64497bab127c016ff1002: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:57:51.591703 7590 consensus_queue.cc:1035] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f61263791f45c5a9c9594389443cde" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 46001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:57:51.592059 7586 consensus_queue.cc:1035] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b32ec0f42c452c940ce2fd9e50c720" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 35203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:57:51.619211 7104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:57:51.619536 7104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:57:51.623162 7104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e3c41cd3a7f47cebb90a9aa6ab98d8b: Sending DeleteTablet for 3 replicas of tablet 64480cde48d6491686b788b73d64e697
I20260926 02:57:51.624496 7496 tablet_service.cc:1514] Processing DeleteTablet for tablet 64480cde48d6491686b788b73d64e697 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:51 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:39152
I20260926 02:57:51.624574 7228 tablet_service.cc:1514] Processing DeleteTablet for tablet 64480cde48d6491686b788b73d64e697 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:51 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:40820
I20260926 02:57:51.625305 7362 tablet_service.cc:1514] Processing DeleteTablet for tablet 64480cde48d6491686b788b73d64e697 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:51 UTC) from {username='kudu', principal='kudu/127.19.185.254@KRBTEST.COM'} at 127.0.0.1:41518
I20260926 02:57:51.626073 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 7165
I20260926 02:57:51.626235 7605 tablet_replica.cc:330] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002: stopping tablet replica
I20260926 02:57:51.626703 7605 raft_consensus.cc:2229] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:57:51.627421 7605 raft_consensus.cc:2258] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:51.634756 7605 ts_tablet_manager.cc:1905] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:51.635667 7604 tablet_replica.cc:330] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720: stopping tablet replica
I20260926 02:57:51.636363 7604 raft_consensus.cc:2229] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:57:51.637066 7604 raft_consensus.cc:2258] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:51.638581 7606 tablet_replica.cc:330] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde: stopping tablet replica
I20260926 02:57:51.638986 7606 raft_consensus.cc:2229] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:57:51.639492 7604 ts_tablet_manager.cc:1905] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:51.639514 7606 raft_consensus.cc:2258] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:51.641603 7606 ts_tablet_manager.cc:1905] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:51.646526 7604 ts_tablet_manager.cc:1918] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:51.646729 7604 log.cc:1196] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/64480cde48d6491686b788b73d64e697
I20260926 02:57:51.647346 7604 ts_tablet_manager.cc:1939] T 64480cde48d6491686b788b73d64e697 P f5b32ec0f42c452c940ce2fd9e50c720: Deleting consensus metadata
I20260926 02:57:51.648658 7606 ts_tablet_manager.cc:1918] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:51.648810 7606 log.cc:1196] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/64480cde48d6491686b788b73d64e697
I20260926 02:57:51.649343 7606 ts_tablet_manager.cc:1939] T 64480cde48d6491686b788b73d64e697 P 21f61263791f45c5a9c9594389443cde: Deleting consensus metadata
I20260926 02:57:51.649459 7090 catalog_manager.cc:4841] TS f5b32ec0f42c452c940ce2fd9e50c720 (127.19.185.193:35203): tablet 64480cde48d6491686b788b73d64e697 (table test-table [id=7037d10dc5764cb286c47d7e0b4b6114]) successfully deleted
I20260926 02:57:51.651151 7088 catalog_manager.cc:4841] TS 21f61263791f45c5a9c9594389443cde (127.19.185.194:46001): tablet 64480cde48d6491686b788b73d64e697 (table test-table [id=7037d10dc5764cb286c47d7e0b4b6114]) successfully deleted
I20260926 02:57:51.653038 7605 ts_tablet_manager.cc:1918] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:51.653216 7605 log.cc:1196] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/64480cde48d6491686b788b73d64e697
I20260926 02:57:51.653741 7605 ts_tablet_manager.cc:1939] T 64480cde48d6491686b788b73d64e697 P 436198a742e64497bab127c016ff1002: Deleting consensus metadata
I20260926 02:57:51.655493 7091 catalog_manager.cc:4841] TS 436198a742e64497bab127c016ff1002 (127.19.185.195:34127): tablet 64480cde48d6491686b788b73d64e697 (table test-table [id=7037d10dc5764cb286c47d7e0b4b6114]) successfully deleted
Sep 26 02:57:51 dist-test-slave-ktng krb5kdc[7059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.193@KRBTEST.COM
I20260926 02:57:51.666100 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.628196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40822 (local address 127.19.185.193:35203)
0926 02:57:51.628441 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:51.628450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:51.650169 (+ 21719us) server_negotiation.cc:407] Connection header received
0926 02:57:51.650345 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:51.650352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:51.650437 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:51.650545 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:51.652446 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.653396 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.654726 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.655130 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.655271 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.660058 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:51.660070 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:51.660085 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:51.660130 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:51.663079 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:51.664037 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:51.664044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:51.664047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:51.664184 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:51.664612 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:51.664615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:51.664617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:51.664996 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:51.665212 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:51.665596 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 02:57:51.665764 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:57:51.666777 7288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:51.815897 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 7165
I20260926 02:57:51.840579 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 7299
Sep 26 02:57:51 dist-test-slave-ktng krb5kdc[7059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.194@KRBTEST.COM
I20260926 02:57:51.858330 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.841421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.19.185.194:46001)
0926 02:57:51.841626 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:51.841633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:51.841704 (+ 71us) server_negotiation.cc:407] Connection header received
0926 02:57:51.841926 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:51.841936 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:51.842003 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:51.842114 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:51.843412 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.844130 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.845375 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.845876 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.846018 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.850710 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:51.850731 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:51.850752 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:51.850804 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:51.854766 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:51.855622 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:51.855629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:51.855632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:51.855840 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:51.856487 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:51.856492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:51.856495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:51.857067 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:51.857399 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:51.857730 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 02:57:51.857975 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:57:51.859025 7422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:51.950297 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 7299
I20260926 02:57:51.968247 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 7433
Sep 26 02:57:51 dist-test-slave-ktng krb5kdc[7059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.195@KRBTEST.COM
I20260926 02:57:51.987510 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:51.969310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.19.185.195:34127)
0926 02:57:51.969560 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:51.969573 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:51.969602 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:57:51.969676 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:51.969682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:51.969796 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:51.969923 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:51.971256 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.972177 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.974173 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:51.974608 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:51.974715 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:51.980647 (+ 5932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:51.980661 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:51.980678 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:51.980721 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:51.984262 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:51.985034 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:51.985052 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:51.985054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:51.985250 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:51.985667 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:51.985674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:51.985678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:51.986076 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:51.986423 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:51.986771 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 02:57:51.987050 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:57:51.988243 7556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:52.112076 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 7433
I20260926 02:57:52.133410 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 7072
I20260926 02:57:52.134543 7134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:52.247977 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 7072
2026-09-26T02:57:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4797 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T02:57:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:57:52Z Disabled control of system clock
I20260926 02:57:52.328991 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:40441
--webserver_interface=127.19.185.254
--webserver_port=0
--builtin_ntp_servers=127.19.185.212:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:40441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:57:52.664064 7615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:52.664367 7615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:52.664520 7615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:52.671881 7615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:57:52.671989 7615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:52.672077 7615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:52.672178 7615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:57:52.672246 7615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:57:52.681540 7615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:40441
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:40441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:52.682847 7615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:52.684748 7615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:52.697489 7623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:52.697527 7621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:52.697808 7620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:52.698437 7615 server_base.cc:1034] running on GCE node
I20260926 02:57:52.699182 7615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:52.700695 7615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:52.701924 7615 hybrid_clock.cc:648] HybridClock initialized: now 1790391472701859 us; error 34 us; skew 500 ppm
I20260926 02:57:52.702301 7615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:57:52.704819 7615 webserver.cc:460] Webserver started at http://127.19.185.254:43071/ using document root <none> and password file <none>
I20260926 02:57:52.705549 7615 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:52.705662 7615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:52.706019 7615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:52.709344 7615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "23e3cca63ecf43c28f7f462080773f80"
format_stamp: "Formatted at 2026-09-26 02:57:52 on dist-test-slave-ktng"
server_key: "61e0cb220f5032e59f040dc8f928c656"
server_key_iv: "a742338b42ca97d186dbd6d9649e0a0a"
server_key_version: "encryptionkey@0"
I20260926 02:57:52.710471 7615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "23e3cca63ecf43c28f7f462080773f80"
format_stamp: "Formatted at 2026-09-26 02:57:52 on dist-test-slave-ktng"
server_key: "61e0cb220f5032e59f040dc8f928c656"
server_key_iv: "a742338b42ca97d186dbd6d9649e0a0a"
server_key_version: "encryptionkey@0"
I20260926 02:57:52.716223 7615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:57:52.719797 7629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:52.721217 7615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 02:57:52.721405 7615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "23e3cca63ecf43c28f7f462080773f80"
format_stamp: "Formatted at 2026-09-26 02:57:52 on dist-test-slave-ktng"
server_key: "61e0cb220f5032e59f040dc8f928c656"
server_key_iv: "a742338b42ca97d186dbd6d9649e0a0a"
server_key_version: "encryptionkey@0"
I20260926 02:57:52.721624 7615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:52.750864 7615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:52.751799 7615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:52.752091 7615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:52.771379 7615 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:40441
I20260926 02:57:52.771433 7680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:40441 every 8 connection(s)
I20260926 02:57:52.772737 7615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:57:52.775660 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 7615
I20260926 02:57:52.775869 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:57:52.776298 20199 external_mini_cluster.cc:1428] Setting key 4bcae108257a18cfb52e27e2d302ec7c
I20260926 02:57:52.776947 7681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:52.790019 7681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80: Bootstrap starting.
I20260926 02:57:52.790741 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:52.778269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.19.185.254:40441)
0926 02:57:52.779267 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:52.779278 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:52.779380 (+ 102us) server_negotiation.cc:407] Connection header received
0926 02:57:52.781770 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:52.781793 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:52.782265 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:52.782708 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:52.783372 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:52.784697 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:52.786031 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:52.786754 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:52.786978 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:52.788309 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:52.788316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:52.788331 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:52.788868 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:52.788951 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:52.788971 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:52.789109 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:52.789340 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 02:57:52.789663 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":378,"threads_started":1}
I20260926 02:57:52.793766 7681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:52.794759 7681 log.cc:824] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:52.797500 7681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80: No bootstrap required, opened a new log
I20260926 02:57:52.805450 7681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e3cca63ecf43c28f7f462080773f80" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40441 } }
I20260926 02:57:52.805848 7681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:52.805955 7681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23e3cca63ecf43c28f7f462080773f80, State: Initialized, Role: FOLLOWER
I20260926 02:57:52.806509 7681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e3cca63ecf43c28f7f462080773f80" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40441 } }
I20260926 02:57:52.806748 7681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:57:52.806895 7681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:57:52.807066 7681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:52.809252 7681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e3cca63ecf43c28f7f462080773f80" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40441 } }
I20260926 02:57:52.809756 7681 leader_election.cc:302] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23e3cca63ecf43c28f7f462080773f80; no voters:
I20260926 02:57:52.810346 7681 leader_election.cc:288] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:57:52.810508 7686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:57:52.811605 7686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [term 1 LEADER]: Becoming Leader. State: Replica: 23e3cca63ecf43c28f7f462080773f80, State: Running, Role: LEADER
I20260926 02:57:52.812192 7686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e3cca63ecf43c28f7f462080773f80" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40441 } }
I20260926 02:57:52.812604 7681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:57:52.815335 7688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23e3cca63ecf43c28f7f462080773f80. Latest consensus state: current_term: 1 leader_uuid: "23e3cca63ecf43c28f7f462080773f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e3cca63ecf43c28f7f462080773f80" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40441 } } }
I20260926 02:57:52.815315 7687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23e3cca63ecf43c28f7f462080773f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23e3cca63ecf43c28f7f462080773f80" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 40441 } } }
I20260926 02:57:52.815848 7688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:52.815902 7687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80 [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:52.824118 7693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:57:52.829658 7693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:57:52.838461 7693 catalog_manager.cc:1347] Generated new cluster ID: 221cbdb2d7c04cde8937792396bfd024
I20260926 02:57:52.838590 7693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:57:52.848551 7693 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:57:52.850315 7693 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:57:52.867231 7693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80: Generated new TSK 0
I20260926 02:57:52.868191 7693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:57:52.883472 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.193:0
--local_ip_for_outbound_sockets=127.19.185.193
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:40441
--builtin_ntp_servers=127.19.185.212:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:57:53.226112 7705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:53.226403 7705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:53.226604 7705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:53.233944 7705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:53.234052 7705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:53.234226 7705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.193
I20260926 02:57:53.243628 7705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.185.193
--webserver_port=0
--tserver_master_addrs=127.19.185.254:40441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.193
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:53.244951 7705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:53.246773 7705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:53.260059 7711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:53.260299 7710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:53.260088 7713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:53.260861 7705 server_base.cc:1034] running on GCE node
I20260926 02:57:53.261536 7705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:53.262578 7705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:53.263818 7705 hybrid_clock.cc:648] HybridClock initialized: now 1790391473263749 us; error 61 us; skew 500 ppm
I20260926 02:57:53.264194 7705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:57:53.266513 7705 webserver.cc:460] Webserver started at http://127.19.185.193:44169/ using document root <none> and password file <none>
I20260926 02:57:53.267418 7705 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:53.267557 7705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:53.267890 7705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:53.270898 7705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/data/instance:
uuid: "704b5daea72243008eac7eaf1611008e"
format_stamp: "Formatted at 2026-09-26 02:57:53 on dist-test-slave-ktng"
server_key: "3cae134be0697f010637146f3513f950"
server_key_iv: "b586791c97ebaa69e429eb0877cb70e9"
server_key_version: "encryptionkey@0"
I20260926 02:57:53.271814 7705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance:
uuid: "704b5daea72243008eac7eaf1611008e"
format_stamp: "Formatted at 2026-09-26 02:57:53 on dist-test-slave-ktng"
server_key: "3cae134be0697f010637146f3513f950"
server_key_iv: "b586791c97ebaa69e429eb0877cb70e9"
server_key_version: "encryptionkey@0"
I20260926 02:57:53.277210 7705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:57:53.280984 7719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:53.282662 7705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:57:53.282871 7705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/wal
uuid: "704b5daea72243008eac7eaf1611008e"
format_stamp: "Formatted at 2026-09-26 02:57:53 on dist-test-slave-ktng"
server_key: "3cae134be0697f010637146f3513f950"
server_key_iv: "b586791c97ebaa69e429eb0877cb70e9"
server_key_version: "encryptionkey@0"
I20260926 02:57:53.283097 7705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:53.307801 7705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:53.308706 7705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:53.309044 7705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:53.310146 7705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:53.311781 7705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:53.311928 7705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:53.312052 7705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:53.312139 7705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:53.347626 7705 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.193:37535
I20260926 02:57:53.347716 7831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.193:37535 every 8 connection(s)
I20260926 02:57:53.349023 7705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/data/info.pb
I20260926 02:57:53.350034 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 7705
I20260926 02:57:53.350252 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/wal/instance
I20260926 02:57:53.350801 20199 external_mini_cluster.cc:1428] Setting key 16843961ca43552b2c1d3e451f39d37a
I20260926 02:57:53.354278 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.194:0
--local_ip_for_outbound_sockets=127.19.185.194
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:40441
--builtin_ntp_servers=127.19.185.212:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:57:53.364154 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:53.352135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.193:52303 (local address 127.19.185.254:40441)
0926 02:57:53.352582 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:53.352586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:53.354191 (+ 1605us) server_negotiation.cc:407] Connection header received
0926 02:57:53.356260 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:53.356269 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:53.356366 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:53.356496 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:53.357664 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:53.358922 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:53.360771 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:53.361508 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:53.361652 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:53.363362 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:53.363367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:53.363371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:53.363391 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:53.363438 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:53.363452 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:53.363526 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:53.363820 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 02:57:53.363902 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":862,"threads_started":1}
I20260926 02:57:53.365679 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:53.352840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:40441 (local address 127.19.185.193:52303)
0926 02:57:53.354011 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:53.354066 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:53.355877 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:53.356703 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:53.356712 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:53.356914 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:53.357485 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:53.357501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:53.359110 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:53.359114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:53.360604 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:53.360617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:53.361857 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:53.361865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:53.362092 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:53.362905 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:57:53.362910 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:53.362967 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:57:53.363085 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:53.363556 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:53.363569 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:53.363580 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 02:57:53.363802 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 02:57:53.364247 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":382,"threads_started":1}
I20260926 02:57:53.367578 7832 heartbeater.cc:347] Connected to a master server at 127.19.185.254:40441
I20260926 02:57:53.367919 7832 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:53.368834 7832 heartbeater.cc:511] Master 127.19.185.254:40441 requested a full tablet report, sending...
I20260926 02:57:53.371057 7646 ts_manager.cc:194] Registered new tserver with Master: 704b5daea72243008eac7eaf1611008e (127.19.185.193:37535)
I20260926 02:57:53.372721 7646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.185.193:52303
W20260926 02:57:53.691166 7836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:53.691499 7836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:53.691635 7836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:53.699043 7836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:53.699138 7836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:53.699354 7836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.194
I20260926 02:57:53.708617 7836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.185.194
--webserver_port=0
--tserver_master_addrs=127.19.185.254:40441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.194
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:53.709919 7836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:53.711756 7836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:53.725819 7842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:53.725842 7841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:53.727598 7844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:53.728861 7836 server_base.cc:1034] running on GCE node
I20260926 02:57:53.729481 7836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:53.730434 7836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:53.731774 7836 hybrid_clock.cc:648] HybridClock initialized: now 1790391473731627 us; error 59 us; skew 500 ppm
I20260926 02:57:53.732158 7836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:57:53.734549 7836 webserver.cc:460] Webserver started at http://127.19.185.194:42971/ using document root <none> and password file <none>
I20260926 02:57:53.735267 7836 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:53.735479 7836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:53.735882 7836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:53.738797 7836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/data/instance:
uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01"
format_stamp: "Formatted at 2026-09-26 02:57:53 on dist-test-slave-ktng"
server_key: "053ae54bac853f77276ff81374293d48"
server_key_iv: "9be40893afd1357e94028df280e01a53"
server_key_version: "encryptionkey@0"
I20260926 02:57:53.739806 7836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance:
uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01"
format_stamp: "Formatted at 2026-09-26 02:57:53 on dist-test-slave-ktng"
server_key: "053ae54bac853f77276ff81374293d48"
server_key_iv: "9be40893afd1357e94028df280e01a53"
server_key_version: "encryptionkey@0"
I20260926 02:57:53.745184 7836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 02:57:53.748657 7850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:53.750155 7836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:57:53.750291 7836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/wal
uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01"
format_stamp: "Formatted at 2026-09-26 02:57:53 on dist-test-slave-ktng"
server_key: "053ae54bac853f77276ff81374293d48"
server_key_iv: "9be40893afd1357e94028df280e01a53"
server_key_version: "encryptionkey@0"
I20260926 02:57:53.750473 7836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:53.767536 7836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:53.768429 7836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:53.768754 7836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:53.769752 7836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:53.771590 7836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:53.771703 7836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:53.771817 7836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:53.771891 7836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:53.813428 7836 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.194:38753
I20260926 02:57:53.813464 7962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.194:38753 every 8 connection(s)
I20260926 02:57:53.814850 7836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/data/info.pb
I20260926 02:57:53.817889 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 7836
I20260926 02:57:53.818078 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/wal/instance
I20260926 02:57:53.818557 20199 external_mini_cluster.cc:1428] Setting key 2f10cf6186af155d0d45d2395e031762
I20260926 02:57:53.821789 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.185.195:0
--local_ip_for_outbound_sockets=127.19.185.195
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:40441
--builtin_ntp_servers=127.19.185.212:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:57:53.830252 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:53.817921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:49055 (local address 127.19.185.254:40441)
0926 02:57:53.818173 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:53.818180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:53.819777 (+ 1597us) server_negotiation.cc:407] Connection header received
0926 02:57:53.822116 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:53.822120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:53.822193 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:53.822279 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:53.823784 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:53.824608 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:53.827211 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:53.827668 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:53.827774 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:53.829454 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:53.829458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:53.829462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:53.829483 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:53.829516 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:53.829528 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:53.829602 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:53.829922 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 02:57:53.830016 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260926 02:57:53.831830 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:53.818404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:40441 (local address 127.19.185.194:49055)
0926 02:57:53.819588 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:53.819650 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:53.821759 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:53.822549 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:53.822564 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:53.822691 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:53.823594 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:53.823622 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:53.824771 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:53.824779 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:53.827039 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:53.827058 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:53.827913 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:53.827923 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:53.828276 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:53.829059 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:57:53.829065 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:53.829120 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:57:53.829236 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:53.829630 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:53.829644 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:53.829655 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 02:57:53.829870 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 02:57:53.830328 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":326,"threads_started":1}
I20260926 02:57:53.833418 7963 heartbeater.cc:347] Connected to a master server at 127.19.185.254:40441
I20260926 02:57:53.833707 7963 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:53.834520 7963 heartbeater.cc:511] Master 127.19.185.254:40441 requested a full tablet report, sending...
I20260926 02:57:53.836139 7646 ts_manager.cc:194] Registered new tserver with Master: 20b5c72a3c2f4905abe7c0e18eaa2b01 (127.19.185.194:38753)
I20260926 02:57:53.837381 7646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.185.194:49055
W20260926 02:57:54.165820 7966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:54.166096 7966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:54.166181 7966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:54.173346 7966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:54.173488 7966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:54.173628 7966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.185.195
I20260926 02:57:54.182685 7966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.185.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.185.195
--webserver_port=0
--tserver_master_addrs=127.19.185.254:40441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.185.195
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:54.183930 7966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:54.185587 7966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:54.198547 7972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:54.198695 7971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:54.198593 7974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:54.199721 7966 server_base.cc:1034] running on GCE node
I20260926 02:57:54.200356 7966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:54.201372 7966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:54.202616 7966 hybrid_clock.cc:648] HybridClock initialized: now 1790391474202574 us; error 54 us; skew 500 ppm
I20260926 02:57:54.203003 7966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:57:54.205401 7966 webserver.cc:460] Webserver started at http://127.19.185.195:45285/ using document root <none> and password file <none>
I20260926 02:57:54.206108 7966 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:54.206238 7966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:54.206550 7966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:54.209425 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/data/instance:
uuid: "6e42fac1de674d718e0b7d249e2ca5cb"
format_stamp: "Formatted at 2026-09-26 02:57:54 on dist-test-slave-ktng"
server_key: "368c70681626352ebfb3657e5b8b8a1a"
server_key_iv: "7930de206e27061561f2afbba45769a5"
server_key_version: "encryptionkey@0"
I20260926 02:57:54.210210 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance:
uuid: "6e42fac1de674d718e0b7d249e2ca5cb"
format_stamp: "Formatted at 2026-09-26 02:57:54 on dist-test-slave-ktng"
server_key: "368c70681626352ebfb3657e5b8b8a1a"
server_key_iv: "7930de206e27061561f2afbba45769a5"
server_key_version: "encryptionkey@0"
I20260926 02:57:54.215409 7966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:57:54.218760 7980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:54.220384 7966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:57:54.220566 7966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/wal
uuid: "6e42fac1de674d718e0b7d249e2ca5cb"
format_stamp: "Formatted at 2026-09-26 02:57:54 on dist-test-slave-ktng"
server_key: "368c70681626352ebfb3657e5b8b8a1a"
server_key_iv: "7930de206e27061561f2afbba45769a5"
server_key_version: "encryptionkey@0"
I20260926 02:57:54.220796 7966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:54.243265 7966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:54.244187 7966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:54.244516 7966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:54.245497 7966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:57:54.247082 7966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:57:54.247215 7966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:54.247422 7966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:57:54.247511 7966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:54.282831 7966 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.195:43177
I20260926 02:57:54.282922 8092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.195:43177 every 8 connection(s)
I20260926 02:57:54.284209 7966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/data/info.pb
I20260926 02:57:54.292668 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 7966
I20260926 02:57:54.292842 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/wal/instance
I20260926 02:57:54.293326 20199 external_mini_cluster.cc:1428] Setting key 1ca65a423c0c1f0495994f5471a1a030
I20260926 02:57:54.297698 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.286848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.195:59861 (local address 127.19.185.254:40441)
0926 02:57:54.287082 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:54.287086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:54.288626 (+ 1540us) server_negotiation.cc:407] Connection header received
0926 02:57:54.290427 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:54.290433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:54.290525 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:54.290637 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:54.291844 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.292897 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.294805 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.295359 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.295501 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.297001 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:54.297005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:54.297008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:54.297025 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:54.297057 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:54.297068 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:54.297140 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:54.297340 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 02:57:54.297435 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260926 02:57:54.298736 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.287577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:40441 (local address 127.19.185.195:59861)
0926 02:57:54.288471 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:54.288511 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:54.290141 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:54.290871 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:54.290879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:54.291027 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:57:54.291680 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:54.291697 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.293051 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.293056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:54.294669 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:54.294681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.295596 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.295602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:54.295886 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.296634 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:57:54.296637 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:57:54.296736 (+ 99us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:57:54.296821 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:57:54.297154 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:57:54.297161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:57:54.297167 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:57:54.297316 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 02:57:54.297616 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":265,"threads_started":1}
I20260926 02:57:54.300118 8093 heartbeater.cc:347] Connected to a master server at 127.19.185.254:40441
I20260926 02:57:54.300477 8093 heartbeater.cc:464] Registering TS with master...
I20260926 02:57:54.301332 8093 heartbeater.cc:511] Master 127.19.185.254:40441 requested a full tablet report, sending...
I20260926 02:57:54.302871 7646 ts_manager.cc:194] Registered new tserver with Master: 6e42fac1de674d718e0b7d249e2ca5cb (127.19.185.195:43177)
I20260926 02:57:54.303789 7646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.185.195:59861
I20260926 02:57:54.308511 20199 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:57:54.323578 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.317405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.19.185.254:40441)
0926 02:57:54.317596 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:54.317600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:54.317968 (+ 368us) server_negotiation.cc:407] Connection header received
0926 02:57:54.318111 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:54.318115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:54.318197 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:54.318293 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:54.318744 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.319641 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.321121 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.321603 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.321711 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.322708 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:54.322713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:54.322716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:54.322735 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:54.322779 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:54.322791 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:54.322874 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:54.323051 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 02:57:54.323132 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:57:54.328225 7646 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:57:54.336057 7646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:57:54.360137 8107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.348991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.194:38753 (local address 127.0.0.1:50818)
0926 02:57:54.351051 (+ 2060us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:54.351094 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:54.351260 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:54.352035 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:54.352043 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:54.352081 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:54.352500 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:54.352515 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.354712 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.354721 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:54.357413 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:54.357426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.358993 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.359003 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:54.359237 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.359412 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 02:57:54.359542 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 02:57:54.359731 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1802,"mutex_wait_us":87,"thread_start_us":1633,"threads_started":1}
I20260926 02:57:54.360164 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.349395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.19.185.194:38753)
0926 02:57:54.349960 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:54.349968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:54.351392 (+ 1424us) server_negotiation.cc:407] Connection header received
0926 02:57:54.351526 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:54.351535 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:54.351798 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:54.352706 (+ 908us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:54.352937 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.354562 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.357632 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.358723 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.358867 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.359085 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:54.359607 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 02:57:54.359788 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":294,"threads_started":1}
I20260926 02:57:54.361406 8106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.349460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.19.185.193:37535)
0926 02:57:54.350553 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:54.350566 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:54.351914 (+ 1348us) server_negotiation.cc:407] Connection header received
0926 02:57:54.352539 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:54.352549 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:54.352772 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:54.352945 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:54.354252 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.355941 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.358589 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.359978 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.360186 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.360414 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:54.360874 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 02:57:54.361066 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":789,"threads_started":1}
I20260926 02:57:54.362244 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.349158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:37535 (local address 127.0.0.1:40838)
0926 02:57:54.351788 (+ 2630us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:54.351806 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:54.352021 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:54.353602 (+ 1581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:54.353607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:54.353632 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:54.354086 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:54.354100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.356062 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.356068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:54.358377 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:54.358392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.360520 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.360524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:54.360645 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.360707 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 02:57:54.360805 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 02:57:54.361976 (+ 1171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2515,"mutex_wait_us":20,"thread_start_us":2394,"threads_started":1}
I20260926 02:57:54.364135 8109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.351919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:43177 (local address 127.0.0.1:33760)
0926 02:57:54.355670 (+ 3751us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:54.355688 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:54.355801 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:54.356368 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:54.356373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:54.356405 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:54.356994 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:54.357007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.358826 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.358834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:54.360927 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:54.360941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.363393 (+ 2452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.363399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:54.363560 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.363591 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:57:54.363693 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 02:57:54.363819 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3629,"mutex_wait_us":45,"thread_start_us":3463,"threads_started":1}
I20260926 02:57:54.364190 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.352394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.19.185.195:43177)
0926 02:57:54.355200 (+ 2806us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:54.355207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:54.355796 (+ 589us) server_negotiation.cc:407] Connection header received
0926 02:57:54.355932 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:54.355939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:54.356161 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:54.356364 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:54.357230 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.358699 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.361135 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.363193 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.363417 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.363611 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:54.363727 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 02:57:54.363876 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2648,"thread_start_us":2445,"threads_started":1}
I20260926 02:57:54.364722 7898 tablet_service.cc:1467] Processing CreateTablet for tablet 1e124b44701947dc93e60e1b72ac05ac (DEFAULT_TABLE table=test-table [id=d1726ad8062942708f7b23e8d88ca030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:54.366199 7898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e124b44701947dc93e60e1b72ac05ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:54.365797 7767 tablet_service.cc:1467] Processing CreateTablet for tablet 1e124b44701947dc93e60e1b72ac05ac (DEFAULT_TABLE table=test-table [id=d1726ad8062942708f7b23e8d88ca030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:54.367089 7767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e124b44701947dc93e60e1b72ac05ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:54.367131 8028 tablet_service.cc:1467] Processing CreateTablet for tablet 1e124b44701947dc93e60e1b72ac05ac (DEFAULT_TABLE table=test-table [id=d1726ad8062942708f7b23e8d88ca030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:57:54.368284 8028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e124b44701947dc93e60e1b72ac05ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:54.376572 7832 heartbeater.cc:503] Master 127.19.185.254:40441 was elected leader, sending a full tablet report...
I20260926 02:57:54.380836 8111 tablet_bootstrap.cc:492] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb: Bootstrap starting.
I20260926 02:57:54.384068 8112 tablet_bootstrap.cc:492] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01: Bootstrap starting.
I20260926 02:57:54.384280 8111 tablet_bootstrap.cc:654] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:54.385645 8111 log.cc:824] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:54.387163 8113 tablet_bootstrap.cc:492] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e: Bootstrap starting.
I20260926 02:57:54.387878 8112 tablet_bootstrap.cc:654] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:54.388926 8111 tablet_bootstrap.cc:492] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb: No bootstrap required, opened a new log
I20260926 02:57:54.389158 8112 log.cc:824] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:54.389207 8111 ts_tablet_manager.cc:1397] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:57:54.391860 8113 tablet_bootstrap.cc:654] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:54.392267 8112 tablet_bootstrap.cc:492] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01: No bootstrap required, opened a new log
I20260926 02:57:54.392521 8112 ts_tablet_manager.cc:1397] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260926 02:57:54.393076 8113 log.cc:824] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:54.396034 8113 tablet_bootstrap.cc:492] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e: No bootstrap required, opened a new log
I20260926 02:57:54.396293 8113 ts_tablet_manager.cc:1397] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 02:57:54.397492 8111 raft_consensus.cc:348] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38753 } } peers { permanent_uuid: "704b5daea72243008eac7eaf1611008e" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37535 } } peers { permanent_uuid: "6e42fac1de674d718e0b7d249e2ca5cb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43177 } }
I20260926 02:57:54.397873 8111 raft_consensus.cc:374] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:54.398037 8111 raft_consensus.cc:729] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e42fac1de674d718e0b7d249e2ca5cb, State: Initialized, Role: FOLLOWER
I20260926 02:57:54.398645 8111 consensus_queue.cc:260] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38753 } } peers { permanent_uuid: "704b5daea72243008eac7eaf1611008e" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37535 } } peers { permanent_uuid: "6e42fac1de674d718e0b7d249e2ca5cb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43177 } }
I20260926 02:57:54.399755 8093 heartbeater.cc:503] Master 127.19.185.254:40441 was elected leader, sending a full tablet report...
I20260926 02:57:54.400383 8111 ts_tablet_manager.cc:1428] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260926 02:57:54.402621 8112 raft_consensus.cc:348] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38753 } } peers { permanent_uuid: "704b5daea72243008eac7eaf1611008e" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37535 } } peers { permanent_uuid: "6e42fac1de674d718e0b7d249e2ca5cb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43177 } }
I20260926 02:57:54.403057 8112 raft_consensus.cc:374] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:54.403151 8112 raft_consensus.cc:729] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b5c72a3c2f4905abe7c0e18eaa2b01, State: Initialized, Role: FOLLOWER
I20260926 02:57:54.403762 8112 consensus_queue.cc:260] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38753 } } peers { permanent_uuid: "704b5daea72243008eac7eaf1611008e" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37535 } } peers { permanent_uuid: "6e42fac1de674d718e0b7d249e2ca5cb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43177 } }
I20260926 02:57:54.404884 7963 heartbeater.cc:503] Master 127.19.185.254:40441 was elected leader, sending a full tablet report...
I20260926 02:57:54.405474 8112 ts_tablet_manager.cc:1428] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260926 02:57:54.407707 8113 raft_consensus.cc:348] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38753 } } peers { permanent_uuid: "704b5daea72243008eac7eaf1611008e" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37535 } } peers { permanent_uuid: "6e42fac1de674d718e0b7d249e2ca5cb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43177 } }
I20260926 02:57:54.408110 8113 raft_consensus.cc:374] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:54.408229 8113 raft_consensus.cc:729] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704b5daea72243008eac7eaf1611008e, State: Initialized, Role: FOLLOWER
I20260926 02:57:54.408860 8113 consensus_queue.cc:260] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38753 } } peers { permanent_uuid: "704b5daea72243008eac7eaf1611008e" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37535 } } peers { permanent_uuid: "6e42fac1de674d718e0b7d249e2ca5cb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43177 } }
I20260926 02:57:54.410475 8113 ts_tablet_manager.cc:1428] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260926 02:57:54.535974 8094 tablet.cc:2369] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:57:54.567225 7964 tablet.cc:2369] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:57:54.600191 8118 raft_consensus.cc:482] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:57:54.600494 8118 raft_consensus.cc:504] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38753 } } peers { permanent_uuid: "704b5daea72243008eac7eaf1611008e" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37535 } } peers { permanent_uuid: "6e42fac1de674d718e0b7d249e2ca5cb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43177 } }
W20260926 02:57:54.601689 7833 tablet.cc:2369] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:57:54.602267 8118 leader_election.cc:288] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 704b5daea72243008eac7eaf1611008e (127.19.185.193:37535), 6e42fac1de674d718e0b7d249e2ca5cb (127.19.185.195:43177)
I20260926 02:57:54.609588 8106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.602685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:60087 (local address 127.19.185.193:37535)
0926 02:57:54.602935 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:54.602945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:54.603407 (+ 462us) server_negotiation.cc:407] Connection header received
0926 02:57:54.603656 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:54.603663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:54.603771 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:54.603869 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:54.604735 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.605576 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.607718 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.608604 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.608727 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.608776 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:54.609201 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 02:57:54.609335 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:57:54.609624 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.603705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.185.194:41009 (local address 127.19.185.195:43177)
0926 02:57:54.603919 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:54.603927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:54.604169 (+ 242us) server_negotiation.cc:407] Connection header received
0926 02:57:54.604313 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:54.604320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:54.604406 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:54.604521 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:57:54.605144 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.606038 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.607965 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.608777 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.608945 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.609017 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:54.609284 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 02:57:54.609406 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:57:54.609691 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.602648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.193:37535 (local address 127.19.185.194:60087)
0926 02:57:54.603246 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:54.603262 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:54.603508 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:54.603979 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:54.603985 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:54.604008 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:54.604527 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:54.604541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.605733 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.605739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:54.607530 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:54.607543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.608805 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.608814 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:54.609007 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.609046 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 02:57:54.609138 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 02:57:54.609269 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":55,"spinlock_wait_cycles":24320,"thread_start_us":351,"threads_started":1}
I20260926 02:57:54.609740 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.603441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.195:43177 (local address 127.19.185.194:41009)
0926 02:57:54.604068 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0926 02:57:54.604081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 02:57:54.604192 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:57:54.604537 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:57:54.604540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:57:54.604558 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:57:54.604973 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:54.604988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.606184 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.606188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:54.607782 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:57:54.607795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.608968 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.608973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:57:54.609097 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.609149 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 02:57:54.609241 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 02:57:54.609358 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":40,"thread_start_us":306,"threads_started":1}
I20260926 02:57:54.610932 8048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e124b44701947dc93e60e1b72ac05ac" candidate_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e42fac1de674d718e0b7d249e2ca5cb" is_pre_election: true
I20260926 02:57:54.610967 7787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e124b44701947dc93e60e1b72ac05ac" candidate_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704b5daea72243008eac7eaf1611008e" is_pre_election: true
I20260926 02:57:54.611577 8048 raft_consensus.cc:2454] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b5c72a3c2f4905abe7c0e18eaa2b01 in term 0.
I20260926 02:57:54.611577 7787 raft_consensus.cc:2454] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b5c72a3c2f4905abe7c0e18eaa2b01 in term 0.
I20260926 02:57:54.612334 7852 leader_election.cc:302] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b5c72a3c2f4905abe7c0e18eaa2b01, 704b5daea72243008eac7eaf1611008e; no voters:
I20260926 02:57:54.612699 8118 raft_consensus.cc:2781] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:57:54.612885 8118 raft_consensus.cc:482] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:57:54.613009 8118 raft_consensus.cc:3037] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:54.615433 8118 raft_consensus.cc:504] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38753 } } peers { permanent_uuid: "704b5daea72243008eac7eaf1611008e" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37535 } } peers { permanent_uuid: "6e42fac1de674d718e0b7d249e2ca5cb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43177 } }
I20260926 02:57:54.616173 8118 leader_election.cc:288] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [CANDIDATE]: Term 1 election: Requested vote from peers 704b5daea72243008eac7eaf1611008e (127.19.185.193:37535), 6e42fac1de674d718e0b7d249e2ca5cb (127.19.185.195:43177)
I20260926 02:57:54.616616 7787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e124b44701947dc93e60e1b72ac05ac" candidate_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704b5daea72243008eac7eaf1611008e"
I20260926 02:57:54.616756 8048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e124b44701947dc93e60e1b72ac05ac" candidate_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e42fac1de674d718e0b7d249e2ca5cb"
I20260926 02:57:54.616819 7787 raft_consensus.cc:3037] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:54.616981 8048 raft_consensus.cc:3037] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:54.619119 7787 raft_consensus.cc:2454] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b5c72a3c2f4905abe7c0e18eaa2b01 in term 1.
I20260926 02:57:54.619181 8048 raft_consensus.cc:2454] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b5c72a3c2f4905abe7c0e18eaa2b01 in term 1.
I20260926 02:57:54.619552 7852 leader_election.cc:302] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b5c72a3c2f4905abe7c0e18eaa2b01, 704b5daea72243008eac7eaf1611008e; no voters:
I20260926 02:57:54.619875 8118 raft_consensus.cc:2781] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:57:54.620393 8118 raft_consensus.cc:686] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [term 1 LEADER]: Becoming Leader. State: Replica: 20b5c72a3c2f4905abe7c0e18eaa2b01, State: Running, Role: LEADER
I20260926 02:57:54.620946 8118 consensus_queue.cc:237] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38753 } } peers { permanent_uuid: "704b5daea72243008eac7eaf1611008e" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37535 } } peers { permanent_uuid: "6e42fac1de674d718e0b7d249e2ca5cb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43177 } }
I20260926 02:57:54.626021 7645 catalog_manager.cc:5495] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b5c72a3c2f4905abe7c0e18eaa2b01 (127.19.185.194). New cstate: current_term: 1 leader_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b5c72a3c2f4905abe7c0e18eaa2b01" member_type: VOTER last_known_addr { host: "127.19.185.194" port: 38753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704b5daea72243008eac7eaf1611008e" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e42fac1de674d718e0b7d249e2ca5cb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43177 } health_report { overall_health: UNKNOWN } } }
I20260926 02:57:54.655682 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.650161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.19.185.194:38753)
0926 02:57:54.650447 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:54.650451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:54.650471 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:57:54.650656 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:54.650660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:54.650755 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:54.650882 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:57:54.651273 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.652099 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.653503 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.653925 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.654014 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.654552 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:57:54.655064 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:57:54.655198 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:54.655250 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 02:57:54.655396 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 02:57:54.664440 7787 raft_consensus.cc:1261] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e [term 1 FOLLOWER]: Refusing update from remote peer 20b5c72a3c2f4905abe7c0e18eaa2b01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:57:54.664463 8048 raft_consensus.cc:1261] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb [term 1 FOLLOWER]: Refusing update from remote peer 20b5c72a3c2f4905abe7c0e18eaa2b01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:57:54.665445 8118 consensus_queue.cc:1035] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704b5daea72243008eac7eaf1611008e" member_type: VOTER last_known_addr { host: "127.19.185.193" port: 37535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:57:54.665745 8122 consensus_queue.cc:1035] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e42fac1de674d718e0b7d249e2ca5cb" member_type: VOTER last_known_addr { host: "127.19.185.195" port: 43177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:57:54.678086 8125 mvcc.cc:205] Tried to move back new op lower bound from 7333443480214761472 to 7333443480051724288. Current Snapshot: MvccSnapshot[applied={T|T < 7333443480214761472}]
I20260926 02:57:54.692570 7645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:57:54.692894 7645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:57:54.696676 7645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23e3cca63ecf43c28f7f462080773f80: Sending DeleteTablet for 3 replicas of tablet 1e124b44701947dc93e60e1b72ac05ac
I20260926 02:57:54.698014 7898 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e124b44701947dc93e60e1b72ac05ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:54 UTC) from {username='slave'} at 127.0.0.1:50818
I20260926 02:57:54.698144 7767 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e124b44701947dc93e60e1b72ac05ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:54 UTC) from {username='slave'} at 127.0.0.1:40838
I20260926 02:57:54.698247 8028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e124b44701947dc93e60e1b72ac05ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:57:54 UTC) from {username='slave'} at 127.0.0.1:33760
I20260926 02:57:54.699630 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 7705
I20260926 02:57:54.711901 8137 tablet_replica.cc:330] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb: stopping tablet replica
I20260926 02:57:54.712595 8137 raft_consensus.cc:2229] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:57:54.712710 8135 tablet_replica.cc:330] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01: stopping tablet replica
I20260926 02:57:54.713140 8135 raft_consensus.cc:2229] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:57:54.713321 8136 tablet_replica.cc:330] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e: stopping tablet replica
I20260926 02:57:54.713515 8137 raft_consensus.cc:2258] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:54.713742 8136 raft_consensus.cc:2229] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:57:54.714232 8136 raft_consensus.cc:2258] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:54.716922 8135 raft_consensus.cc:2258] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:57:54.717440 8136 ts_tablet_manager.cc:1905] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:54.718016 8137 ts_tablet_manager.cc:1905] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:54.721511 8135 ts_tablet_manager.cc:1905] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:57:54.725720 8137 ts_tablet_manager.cc:1918] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:54.725932 8137 log.cc:1196] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-2/wal/wals/1e124b44701947dc93e60e1b72ac05ac
I20260926 02:57:54.726589 8137 ts_tablet_manager.cc:1939] T 1e124b44701947dc93e60e1b72ac05ac P 6e42fac1de674d718e0b7d249e2ca5cb: Deleting consensus metadata
I20260926 02:57:54.726591 8136 ts_tablet_manager.cc:1918] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:54.726740 8136 log.cc:1196] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-0/wal/wals/1e124b44701947dc93e60e1b72ac05ac
I20260926 02:57:54.727273 8136 ts_tablet_manager.cc:1939] T 1e124b44701947dc93e60e1b72ac05ac P 704b5daea72243008eac7eaf1611008e: Deleting consensus metadata
I20260926 02:57:54.728360 8106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.700962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40842 (local address 127.19.185.193:37535)
0926 02:57:54.701202 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:54.701212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:54.708529 (+ 7317us) server_negotiation.cc:407] Connection header received
0926 02:57:54.709620 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:54.709631 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:54.709822 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:54.709942 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:54.711620 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.713100 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.715414 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.716749 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.716879 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.725926 (+ 9047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:54.725944 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:54.725965 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:54.726876 (+ 911us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:54.726951 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:54.726991 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:54.727114 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:54.727787 (+ 673us) server_negotiation.cc:299] Negotiation successful
0926 02:57:54.727939 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:57:54.729009 7631 catalog_manager.cc:4841] TS 704b5daea72243008eac7eaf1611008e (127.19.185.193:37535): tablet 1e124b44701947dc93e60e1b72ac05ac (table test-table [id=d1726ad8062942708f7b23e8d88ca030]) successfully deleted
I20260926 02:57:54.729328 7631 catalog_manager.cc:4841] TS 6e42fac1de674d718e0b7d249e2ca5cb (127.19.185.195:43177): tablet 1e124b44701947dc93e60e1b72ac05ac (table test-table [id=d1726ad8062942708f7b23e8d88ca030]) successfully deleted
I20260926 02:57:54.729673 7827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:54.733826 8135 ts_tablet_manager.cc:1918] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:57:54.734009 8135 log.cc:1196] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01: Deleting WAL directory at /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391151502216-20199-0/minicluster-data/ts-1/wal/wals/1e124b44701947dc93e60e1b72ac05ac
I20260926 02:57:54.734534 8135 ts_tablet_manager.cc:1939] T 1e124b44701947dc93e60e1b72ac05ac P 20b5c72a3c2f4905abe7c0e18eaa2b01: Deleting consensus metadata
I20260926 02:57:54.736325 7630 catalog_manager.cc:4841] TS 20b5c72a3c2f4905abe7c0e18eaa2b01 (127.19.185.194:38753): tablet 1e124b44701947dc93e60e1b72ac05ac (table test-table [id=d1726ad8062942708f7b23e8d88ca030]) successfully deleted
I20260926 02:57:54.826264 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 7705
I20260926 02:57:54.851035 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 7836
I20260926 02:57:54.859920 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.852055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.19.185.194:38753)
0926 02:57:54.852242 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:54.852248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:54.852273 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:57:54.852506 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:54.852512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:54.852605 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:54.852730 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:54.853240 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.854505 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.856676 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.857094 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.857195 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.858446 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:54.858455 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:54.858468 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:54.859070 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:54.859120 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:54.859139 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:54.859219 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:54.859450 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 02:57:54.859537 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:57:54.860715 7958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:54.972971 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 7836
I20260926 02:57:54.992568 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 7966
I20260926 02:57:54.999871 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:54.993401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.19.185.195:43177)
0926 02:57:54.993631 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:54.993638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:54.993752 (+ 114us) server_negotiation.cc:407] Connection header received
0926 02:57:54.994026 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:54.994030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:54.994124 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:54.994212 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:54.994666 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.995482 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.996770 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:54.997163 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:54.997261 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:54.998285 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:54.998294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:54.998308 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:54.998899 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:54.998957 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:54.998977 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:54.999066 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:54.999251 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 02:57:54.999464 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 02:57:55.000551 8088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:55.090991 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 7966
I20260926 02:57:55.108088 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 7615
I20260926 02:57:55.109154 7676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:55.225425 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 7615
2026-09-26T02:57:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2976 ms)
[----------] 6 tests from EncryptionPolicyTest (22521 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:57:55 dist-test-slave-ktng krb5kdc[8145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:57:55 dist-test-slave-ktng krb5kdc[8145](info): set up 2 sockets
Sep 26 02:57:55 dist-test-slave-ktng krb5kdc[8145](info): commencing operation
krb5kdc: starting...
W20260926 02:57:56.365442 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:57:56 dist-test-slave-ktng krb5kdc[8145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:57:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:57:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:57:56.637192 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45925
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:56.976267 8158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:56.976549 8158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:56.976708 8158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:56.984385 8158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:57:56.984499 8158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:56.984598 8158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:56.984714 8158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:57:56.984795 8158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:57:56.996024 8158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45925
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:56.997581 8158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:56.999126 8158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:57.013041 8164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:57.013015 8166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:57.013103 8163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:57.013862 8158 server_base.cc:1034] running on GCE node
I20260926 02:57:57.014744 8158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:57.016429 8158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:57.017815 8158 hybrid_clock.cc:648] HybridClock initialized: now 1790391477017776 us; error 55 us; skew 500 ppm
I20260926 02:57:57.018220 8158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:57 dist-test-slave-ktng krb5kdc[8145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:57.027770 8158 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:57:57.029609 8158 webserver.cc:460] Webserver started at http://127.0.0.1:43135/ using document root <none> and password file <none>
I20260926 02:57:57.030478 8158 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:57.030655 8158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:57.031016 8158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:57.034574 8158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "86ffb481c0214964bb0ac198920d1f71"
format_stamp: "Formatted at 2026-09-26 02:57:57 on dist-test-slave-ktng"
server_key: "f958b71010342b3218826e02372850c9"
server_key_iv: "62868760cc3d0118302a32b2f7920865"
server_key_version: "encryptionkey@0"
I20260926 02:57:57.035537 8158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "86ffb481c0214964bb0ac198920d1f71"
format_stamp: "Formatted at 2026-09-26 02:57:57 on dist-test-slave-ktng"
server_key: "f958b71010342b3218826e02372850c9"
server_key_iv: "62868760cc3d0118302a32b2f7920865"
server_key_version: "encryptionkey@0"
I20260926 02:57:57.041621 8158 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:57:57.045744 8173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:57.047461 8158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:57:57.047663 8158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "86ffb481c0214964bb0ac198920d1f71"
format_stamp: "Formatted at 2026-09-26 02:57:57 on dist-test-slave-ktng"
server_key: "f958b71010342b3218826e02372850c9"
server_key_iv: "62868760cc3d0118302a32b2f7920865"
server_key_version: "encryptionkey@0"
I20260926 02:57:57.047886 8158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:57.063711 8158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:57.068426 8158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:57.068722 8158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:57.087584 8158 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45925
I20260926 02:57:57.087618 8224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45925 every 8 connection(s)
I20260926 02:57:57.088963 8158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:57:57.092701 8225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:57.092700 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 8158
I20260926 02:57:57.092922 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:57:57.093576 20199 external_mini_cluster.cc:1428] Setting key d3729d3a3a1e011832a844281d027ae3
I20260926 02:57:57.107133 8225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71: Bootstrap starting.
Sep 26 02:57:57 dist-test-slave-ktng krb5kdc[8145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:57:57.110230 8225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:57.111671 8225 log.cc:824] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:57.115386 8225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71: No bootstrap required, opened a new log
I20260926 02:57:57.119726 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:57.095774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37573 (local address 127.0.0.1:45925)
0926 02:57:57.096580 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:57.096596 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:57.096648 (+ 52us) server_negotiation.cc:407] Connection header received
0926 02:57:57.099532 (+ 2884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:57.099560 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:57.100093 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:57.100533 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:57.101843 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:57.103909 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:57.105179 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:57.106191 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:57.106390 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:57.110428 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:57.110440 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:57.110460 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:57.110511 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:57.114807 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:57.115730 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:57.115738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:57.115746 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:57.115988 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:57.116473 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:57.116478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:57.116481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:57.117163 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:57.117505 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:57.117942 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 02:57:57.118313 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":328,"threads_started":1}
I20260926 02:57:57.124135 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:57:57.125418 8225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ffb481c0214964bb0ac198920d1f71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } }
I20260926 02:57:57.125787 8225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:57.125901 8225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ffb481c0214964bb0ac198920d1f71, State: Initialized, Role: FOLLOWER
I20260926 02:57:57.126469 8225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ffb481c0214964bb0ac198920d1f71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } }
I20260926 02:57:57.126717 8225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:57:57.126843 8225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:57:57.127022 8225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:57.129446 8225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ffb481c0214964bb0ac198920d1f71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } }
I20260926 02:57:57.130080 8225 leader_election.cc:302] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86ffb481c0214964bb0ac198920d1f71; no voters:
I20260926 02:57:57.130856 8225 leader_election.cc:288] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:57:57.130932 8235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:57:57.133285 8225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:57:57.135864 8235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [term 1 LEADER]: Becoming Leader. State: Replica: 86ffb481c0214964bb0ac198920d1f71, State: Running, Role: LEADER
I20260926 02:57:57.136600 8235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ffb481c0214964bb0ac198920d1f71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } }
I20260926 02:57:57.139659 8237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86ffb481c0214964bb0ac198920d1f71. Latest consensus state: current_term: 1 leader_uuid: "86ffb481c0214964bb0ac198920d1f71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ffb481c0214964bb0ac198920d1f71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } } }
I20260926 02:57:57.140062 8237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:57.140625 8236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86ffb481c0214964bb0ac198920d1f71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ffb481c0214964bb0ac198920d1f71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } } }
I20260926 02:57:57.140955 8236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71 [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:57.145123 8241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:57:57.150488 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:57.129282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59379 (local address 127.0.0.1:45925)
0926 02:57:57.129527 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:57.129537 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:57.129930 (+ 393us) server_negotiation.cc:407] Connection header received
0926 02:57:57.130132 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:57.130136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:57.130196 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:57.130317 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:57.132330 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:57.133483 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:57.135629 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:57.136408 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:57.136554 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:57.140273 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:57.140279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:57.140284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:57:57.140343 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:57.145472 (+ 5129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:57.146762 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:57.146771 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:57.146777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:57.147075 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:57:57.147826 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:57:57.147830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:57:57.147838 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:57:57.148298 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:57.149691 (+ 1393us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:57.149885 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 02:57:57.150067 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:57:57.150578 8241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:57:57.159428 8241 catalog_manager.cc:1347] Generated new cluster ID: 10880a3e202247c0a74bc43dd9b8cfb6
I20260926 02:57:57.159547 8241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:57:57.174561 8241 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:57:57.176190 8241 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:57:57.183919 8241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86ffb481c0214964bb0ac198920d1f71: Generated new TSK 0
I20260926 02:57:57.184749 8241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:57:57.190727 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 8158
I20260926 02:57:57.191972 8220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:57:57.280627 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 8158
W20260926 02:57:57.295483 8143 connection.cc:535] client connection to 127.0.0.1:45925 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45925): Connection reset by peer (error 104)
2026-09-26T02:57:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2045 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:57:57 dist-test-slave-ktng krb5kdc[8258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:57:57 dist-test-slave-ktng krb5kdc[8258](info): set up 2 sockets
Sep 26 02:57:57 dist-test-slave-ktng krb5kdc[8258](info): commencing operation
krb5kdc: starting...
W20260926 02:57:58.401188 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:57:58 dist-test-slave-ktng krb5kdc[8258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:57:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:57:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:57:58.679234 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33905
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:57:59.019585 8271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:57:59.019881 8271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:57:59.019979 8271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:57:59.027070 8271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:57:59.027176 8271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:57:59.027222 8271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:57:59.027288 8271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:57:59.027349 8271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:57:59.037247 8271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:57:59.038545 8271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:57:59.039825 8271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:57:59.052408 8279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:59.052466 8276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:57:59.053022 8277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:57:59.057035 8271 server_base.cc:1034] running on GCE node
I20260926 02:57:59.057914 8271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:57:59.059540 8271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:57:59.060783 8271 hybrid_clock.cc:648] HybridClock initialized: now 1790391479060724 us; error 60 us; skew 500 ppm
I20260926 02:57:59.061152 8271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:57:59 dist-test-slave-ktng krb5kdc[8258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:57:59.064487 8271 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:57:59.066154 8271 webserver.cc:460] Webserver started at http://127.0.0.1:42183/ using document root <none> and password file <none>
I20260926 02:57:59.066870 8271 fs_manager.cc:359] Metadata directory not provided
I20260926 02:57:59.067027 8271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:57:59.067425 8271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:57:59.070582 8271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "77f79536e99b45249eda9b39b93d56eb"
format_stamp: "Formatted at 2026-09-26 02:57:59 on dist-test-slave-ktng"
server_key: "98106ccaa2fa6348d1dfa1df69e27d9d"
server_key_iv: "03d3446562754535d3a7919da8cf6cb2"
server_key_version: "encryptionkey@0"
I20260926 02:57:59.071468 8271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "77f79536e99b45249eda9b39b93d56eb"
format_stamp: "Formatted at 2026-09-26 02:57:59 on dist-test-slave-ktng"
server_key: "98106ccaa2fa6348d1dfa1df69e27d9d"
server_key_iv: "03d3446562754535d3a7919da8cf6cb2"
server_key_version: "encryptionkey@0"
I20260926 02:57:59.076773 8271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:57:59.080456 8286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:57:59.081827 8271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:57:59.082026 8271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "77f79536e99b45249eda9b39b93d56eb"
format_stamp: "Formatted at 2026-09-26 02:57:59 on dist-test-slave-ktng"
server_key: "98106ccaa2fa6348d1dfa1df69e27d9d"
server_key_iv: "03d3446562754535d3a7919da8cf6cb2"
server_key_version: "encryptionkey@0"
I20260926 02:57:59.082229 8271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:57:59.110584 8271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:57:59.114934 8271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:57:59.115239 8271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:57:59.134367 8271 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33905
I20260926 02:57:59.134413 8337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33905 every 8 connection(s)
I20260926 02:57:59.135913 8271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:57:59.136997 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 8271
I20260926 02:57:59.137246 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:57:59.137816 20199 external_mini_cluster.cc:1428] Setting key b23a46e088d04962fbf58bf543c857b7
I20260926 02:57:59.140901 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:57:59.150811 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:59.139728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38237 (local address 127.0.0.1:33905)
0926 02:57:59.140492 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:59.140504 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:59.140545 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:57:59.142215 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:59.142238 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:59.142649 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:59.143043 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:59.143652 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:59.144959 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:59.146292 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:59.146969 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:59.147202 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:59.148442 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:59.148450 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:59.148463 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:59.149016 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:59.149086 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:59.149113 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:59.149283 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:59.149570 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 02:57:59.149738 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":330,"threads_started":1}
I20260926 02:57:59.152392 8338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb: Bootstrap starting.
I20260926 02:57:59.155022 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:57:59.155907 8338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb: Neither blocks nor log segments found. Creating new log.
I20260926 02:57:59.156908 8338 log.cc:824] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb: Log is configured to *not* fsync() on all Append() calls
I20260926 02:57:59.160488 8338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb: No bootstrap required, opened a new log
I20260926 02:57:59.169202 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:57:59.159528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40731 (local address 127.0.0.1:33905)
0926 02:57:59.159736 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:57:59.159743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:57:59.160130 (+ 387us) server_negotiation.cc:407] Connection header received
0926 02:57:59.160480 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:57:59.160487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:57:59.160580 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:57:59.160696 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:57:59.161589 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:59.163033 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:59.165224 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:57:59.166049 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:57:59.166188 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:57:59.167726 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:57:59.167732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:57:59.167736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:57:59.167764 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:57:59.167826 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:57:59.167850 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:57:59.167979 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 02:57:59.168212 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 02:57:59.168827 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:57:59.172165 8338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f79536e99b45249eda9b39b93d56eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } }
I20260926 02:57:59.172597 8338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:57:59.172736 8338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77f79536e99b45249eda9b39b93d56eb, State: Initialized, Role: FOLLOWER
I20260926 02:57:59.173334 8338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f79536e99b45249eda9b39b93d56eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } }
I20260926 02:57:59.173593 8338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:57:59.173758 8338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:57:59.173955 8338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:57:59.176374 8338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f79536e99b45249eda9b39b93d56eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } }
I20260926 02:57:59.176916 8338 leader_election.cc:302] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77f79536e99b45249eda9b39b93d56eb; no voters:
I20260926 02:57:59.177619 8338 leader_election.cc:288] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:57:59.177755 8348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:57:59.178578 8348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [term 1 LEADER]: Becoming Leader. State: Replica: 77f79536e99b45249eda9b39b93d56eb, State: Running, Role: LEADER
I20260926 02:57:59.179183 8348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f79536e99b45249eda9b39b93d56eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } }
I20260926 02:57:59.180079 8338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:57:59.182123 8349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77f79536e99b45249eda9b39b93d56eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f79536e99b45249eda9b39b93d56eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } } }
I20260926 02:57:59.182225 8350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77f79536e99b45249eda9b39b93d56eb. Latest consensus state: current_term: 1 leader_uuid: "77f79536e99b45249eda9b39b93d56eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f79536e99b45249eda9b39b93d56eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } } }
I20260926 02:57:59.182569 8349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:59.182601 8350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb [sys.catalog]: This master's current role is: LEADER
I20260926 02:57:59.185350 8355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:57:59.190824 8355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:57:59.200815 8355 catalog_manager.cc:1347] Generated new cluster ID: f19987762cd34068b96f3b78a417fc49
I20260926 02:57:59.200940 8355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:57:59.224279 8355 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:57:59.226195 8355 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:57:59.238047 8355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77f79536e99b45249eda9b39b93d56eb: Generated new TSK 0
I20260926 02:57:59.239182 8355 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:57:59.251147 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 8271
W20260926 02:57:59.252570 8333 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38237
W20260926 02:57:59.253492 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 8271: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:57:59.253705 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 8271
W20260926 02:57:59.276701 8256 connection.cc:535] client connection to 127.0.0.1:33905 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33905): Connection reset by peer (error 104)
2026-09-26T02:57:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1980 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:57:59 dist-test-slave-ktng krb5kdc[8369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:57:59 dist-test-slave-ktng krb5kdc[8369](info): set up 2 sockets
Sep 26 02:57:59 dist-test-slave-ktng krb5kdc[8369](info): commencing operation
krb5kdc: starting...
W20260926 02:58:00.420670 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:00 dist-test-slave-ktng krb5kdc[8369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:58:00.687116 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:01.026096 8382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:01.026423 8382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:01.026667 8382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:01.034819 8382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:01.034955 8382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:01.035084 8382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:01.035233 8382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:01.035339 8382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:01.046638 8382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:01.048079 8382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:01.049486 8382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:01.062666 8388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:01.062698 8387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:01.062628 8390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:01.063371 8382 server_base.cc:1034] running on GCE node
I20260926 02:58:01.064224 8382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:01.065711 8382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:01.067003 8382 hybrid_clock.cc:648] HybridClock initialized: now 1790391481066929 us; error 57 us; skew 500 ppm
I20260926 02:58:01.067481 8382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:01 dist-test-slave-ktng krb5kdc[8369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:01.071080 8382 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:58:01.072772 8382 webserver.cc:460] Webserver started at http://127.0.0.1:38423/ using document root <none> and password file <none>
I20260926 02:58:01.073472 8382 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:01.073621 8382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:01.073947 8382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:01.077159 8382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "9d5e256ca4ce4373a022be9e5689e71c"
format_stamp: "Formatted at 2026-09-26 02:58:01 on dist-test-slave-ktng"
server_key: "de1211385a22060059db0159836adae8"
server_key_iv: "45b5abfc32911cf7264310265df2fc68"
server_key_version: "encryptionkey@0"
I20260926 02:58:01.077976 8382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "9d5e256ca4ce4373a022be9e5689e71c"
format_stamp: "Formatted at 2026-09-26 02:58:01 on dist-test-slave-ktng"
server_key: "de1211385a22060059db0159836adae8"
server_key_iv: "45b5abfc32911cf7264310265df2fc68"
server_key_version: "encryptionkey@0"
I20260926 02:58:01.083547 8382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:58:01.087193 8397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:01.088817 8382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:58:01.089020 8382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "9d5e256ca4ce4373a022be9e5689e71c"
format_stamp: "Formatted at 2026-09-26 02:58:01 on dist-test-slave-ktng"
server_key: "de1211385a22060059db0159836adae8"
server_key_iv: "45b5abfc32911cf7264310265df2fc68"
server_key_version: "encryptionkey@0"
I20260926 02:58:01.089246 8382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:01.095712 8382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:01.100385 8382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:01.100697 8382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:01.121578 8382 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39619
I20260926 02:58:01.121625 8448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39619 every 8 connection(s)
I20260926 02:58:01.123066 8382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:01.123607 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 8382
I20260926 02:58:01.123826 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:01.124521 20199 external_mini_cluster.cc:1428] Setting key f4383b1270082c2a73f12b73a940f0c2
I20260926 02:58:01.127575 8449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:01.132581 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:01.126453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48325 (local address 127.0.0.1:39619)
0926 02:58:01.127268 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:01.127282 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:01.127375 (+ 93us) server_negotiation.cc:407] Connection header received
0926 02:58:01.129021 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:01.129043 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:01.129382 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:01.129788 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:01.130895 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:01.130901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:01.130914 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:01.130928 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:01.131014 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:01.131034 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:01.131317 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:01.131459 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 02:58:01.131629 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":358,"threads_started":1}
I20260926 02:58:01.136988 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:01.140131 8449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c: Bootstrap starting.
I20260926 02:58:01.143237 8449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:01.144408 8449 log.cc:824] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:01.147600 8449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c: No bootstrap required, opened a new log
I20260926 02:58:01.148330 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:01.145434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49207 (local address 127.0.0.1:39619)
0926 02:58:01.145672 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:01.145682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:01.146161 (+ 479us) server_negotiation.cc:407] Connection header received
0926 02:58:01.146366 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:01.146376 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:01.146425 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:01.146530 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:01.147638 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:01.147643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:01.147646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:01.147663 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:01.147729 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:01.147751 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:01.147880 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:01.147982 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 02:58:01.148054 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 02:58:01.155647 8449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5e256ca4ce4373a022be9e5689e71c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39619 } }
I20260926 02:58:01.156095 8449 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:01.156255 8449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d5e256ca4ce4373a022be9e5689e71c, State: Initialized, Role: FOLLOWER
I20260926 02:58:01.156888 8449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5e256ca4ce4373a022be9e5689e71c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39619 } }
I20260926 02:58:01.157097 8449 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:01.157197 8449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:01.157320 8449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:01.159461 8449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5e256ca4ce4373a022be9e5689e71c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39619 } }
I20260926 02:58:01.159951 8449 leader_election.cc:302] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d5e256ca4ce4373a022be9e5689e71c; no voters:
I20260926 02:58:01.160617 8449 leader_election.cc:288] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:01.161120 8459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:01.162120 8459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [term 1 LEADER]: Becoming Leader. State: Replica: 9d5e256ca4ce4373a022be9e5689e71c, State: Running, Role: LEADER
I20260926 02:58:01.162891 8459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5e256ca4ce4373a022be9e5689e71c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39619 } }
I20260926 02:58:01.166626 8461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d5e256ca4ce4373a022be9e5689e71c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5e256ca4ce4373a022be9e5689e71c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39619 } } }
I20260926 02:58:01.167006 8461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:01.167148 8449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:01.171657 8463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:01.171581 8460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d5e256ca4ce4373a022be9e5689e71c. Latest consensus state: current_term: 1 leader_uuid: "9d5e256ca4ce4373a022be9e5689e71c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5e256ca4ce4373a022be9e5689e71c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39619 } } }
I20260926 02:58:01.171936 8460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:01.177608 8463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:01.189163 8463 catalog_manager.cc:1347] Generated new cluster ID: 7d5a81bd8a2e44f99ebea93e6e81305b
I20260926 02:58:01.189304 8463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:01.215137 8463 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:01.215699 8463 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:01.226812 8463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d5e256ca4ce4373a022be9e5689e71c: Generated new TSK 0
I20260926 02:58:01.227939 8463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:01.236150 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 8382
W20260926 02:58:01.237524 8444 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48325
W20260926 02:58:01.238292 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 8382: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:58:01.238438 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 8382
2026-09-26T02:58:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1979 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:01 dist-test-slave-ktng krb5kdc[8481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:01 dist-test-slave-ktng krb5kdc[8481](info): set up 2 sockets
Sep 26 02:58:01 dist-test-slave-ktng krb5kdc[8481](info): commencing operation
krb5kdc: starting...
W20260926 02:58:02.368825 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:02 dist-test-slave-ktng krb5kdc[8481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:58:02.635254 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:02.968523 8494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:02.968855 8494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:02.969026 8494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:02.976600 8494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:02.976719 8494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:02.976831 8494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:02.976948 8494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:02.977032 8494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:02.986951 8494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:02.988379 8494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:02.989768 8494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:03.002957 8499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:03.002972 8500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:03.002933 8502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:03.003839 8494 server_base.cc:1034] running on GCE node
I20260926 02:58:03.004721 8494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:03.006191 8494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:03.007414 8494 hybrid_clock.cc:648] HybridClock initialized: now 1790391483007369 us; error 29 us; skew 500 ppm
I20260926 02:58:03.007880 8494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:03 dist-test-slave-ktng krb5kdc[8481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:03.016036 8494 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:58:03.017694 8494 webserver.cc:460] Webserver started at http://127.0.0.1:35445/ using document root <none> and password file <none>
I20260926 02:58:03.018466 8494 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:03.018618 8494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:03.018944 8494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:03.022014 8494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "bd84d8f0bd554901b419304a1173e504"
format_stamp: "Formatted at 2026-09-26 02:58:03 on dist-test-slave-ktng"
server_key: "3b55148966672ebe5b0b239c69587c6f"
server_key_iv: "5ef0e91931ca041ee118c68f684c1cf7"
server_key_version: "encryptionkey@0"
I20260926 02:58:03.022789 8494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "bd84d8f0bd554901b419304a1173e504"
format_stamp: "Formatted at 2026-09-26 02:58:03 on dist-test-slave-ktng"
server_key: "3b55148966672ebe5b0b239c69587c6f"
server_key_iv: "5ef0e91931ca041ee118c68f684c1cf7"
server_key_version: "encryptionkey@0"
I20260926 02:58:03.028172 8494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 02:58:03.031997 8509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:03.033413 8494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 02:58:03.033607 8494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "bd84d8f0bd554901b419304a1173e504"
format_stamp: "Formatted at 2026-09-26 02:58:03 on dist-test-slave-ktng"
server_key: "3b55148966672ebe5b0b239c69587c6f"
server_key_iv: "5ef0e91931ca041ee118c68f684c1cf7"
server_key_version: "encryptionkey@0"
I20260926 02:58:03.033825 8494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:03.050622 8494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:03.055456 8494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:03.055831 8494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:03.077505 8494 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41385
I20260926 02:58:03.077520 8560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41385 every 8 connection(s)
I20260926 02:58:03.079093 8494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:03.080935 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 8494
I20260926 02:58:03.081166 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:03.081871 20199 external_mini_cluster.cc:1428] Setting key 117f3ea34c4d0494712109b643725645
I20260926 02:58:03.084066 8561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:58:03 dist-test-slave-ktng krb5kdc[8481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:58:03.097940 8561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504: Bootstrap starting.
I20260926 02:58:03.101094 8561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:03.102175 8561 log.cc:824] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:03.104928 8561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504: No bootstrap required, opened a new log
I20260926 02:58:03.106145 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:03.083905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47583 (local address 127.0.0.1:41385)
0926 02:58:03.084876 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:03.084889 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:03.084933 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:58:03.086753 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:03.086777 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:03.087250 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:03.087684 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:03.089568 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:03.091183 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:03.092936 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:03.093668 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:03.093859 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:03.098766 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:03.098774 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:03.098789 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:03.098829 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:03.102047 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:03.102860 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:03.102868 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:03.102874 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:03.103045 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:03.103510 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:03.103517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:03.103519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:03.104013 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:03.104328 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:03.104762 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 02:58:03.105200 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":508,"threads_started":1}
I20260926 02:58:03.109885 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:03.113030 8561 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd84d8f0bd554901b419304a1173e504" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41385 } }
I20260926 02:58:03.113404 8561 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:03.113509 8561 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd84d8f0bd554901b419304a1173e504, State: Initialized, Role: FOLLOWER
I20260926 02:58:03.114044 8561 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd84d8f0bd554901b419304a1173e504" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41385 } }
I20260926 02:58:03.114313 8561 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:03.114465 8561 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:03.114662 8561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:03.117057 8561 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd84d8f0bd554901b419304a1173e504" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41385 } }
I20260926 02:58:03.117581 8561 leader_election.cc:302] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd84d8f0bd554901b419304a1173e504; no voters:
I20260926 02:58:03.118325 8561 leader_election.cc:288] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:03.118422 8571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:03.119688 8571 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [term 1 LEADER]: Becoming Leader. State: Replica: bd84d8f0bd554901b419304a1173e504, State: Running, Role: LEADER
I20260926 02:58:03.120375 8571 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd84d8f0bd554901b419304a1173e504" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41385 } }
I20260926 02:58:03.120857 8561 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:03.123487 8573 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd84d8f0bd554901b419304a1173e504. Latest consensus state: current_term: 1 leader_uuid: "bd84d8f0bd554901b419304a1173e504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd84d8f0bd554901b419304a1173e504" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41385 } } }
I20260926 02:58:03.124320 8573 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:03.123886 8572 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd84d8f0bd554901b419304a1173e504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd84d8f0bd554901b419304a1173e504" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41385 } } }
I20260926 02:58:03.124982 8572 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:03.130622 8579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:03.133248 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:03.114385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57395 (local address 127.0.0.1:41385)
0926 02:58:03.114583 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:03.114591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:03.115109 (+ 518us) server_negotiation.cc:407] Connection header received
0926 02:58:03.115334 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:03.115340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:03.115418 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:03.115535 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:03.116949 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:03.117825 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:03.119476 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:03.120143 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:03.120299 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:03.123485 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:03.123491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:03.123499 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:03.123544 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:03.129933 (+ 6389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:03.130919 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:03.130927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:03.130934 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:03.131146 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:03.131628 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:03.131633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:03.131636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:03.132016 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:03.132286 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:03.132652 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 02:58:03.132876 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:58:03.140285 8579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:03.149410 8579 catalog_manager.cc:1347] Generated new cluster ID: ddb6c386d0ba43c0a4761506339b68e8
I20260926 02:58:03.149523 8579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:03.169157 8579 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:03.171037 8579 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:03.181283 8579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd84d8f0bd554901b419304a1173e504: Generated new TSK 0
I20260926 02:58:03.182322 8579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:03.192541 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 8494
I20260926 02:58:03.193647 8556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:58:03.286569 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 8494
2026-09-26T02:58:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:03 dist-test-slave-ktng krb5kdc[8593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:03 dist-test-slave-ktng krb5kdc[8593](info): set up 2 sockets
Sep 26 02:58:03 dist-test-slave-ktng krb5kdc[8593](info): commencing operation
krb5kdc: starting...
W20260926 02:58:04.436664 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:04 dist-test-slave-ktng krb5kdc[8593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:58:04.710529 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:05.048815 8606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:05.049122 8606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:05.049213 8606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:05.056742 8606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:05.056869 8606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:05.056939 8606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:05.057010 8606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:05.057060 8606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:05.067375 8606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:05.068678 8606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:05.069962 8606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:05.082261 8612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:05.082334 8611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:05.082298 8614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:05.083237 8606 server_base.cc:1034] running on GCE node
I20260926 02:58:05.084151 8606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:05.085736 8606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:05.086967 8606 hybrid_clock.cc:648] HybridClock initialized: now 1790391485086917 us; error 36 us; skew 500 ppm
I20260926 02:58:05.087441 8606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:05 dist-test-slave-ktng krb5kdc[8593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:05.090981 8606 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:58:05.092693 8606 webserver.cc:460] Webserver started at http://127.0.0.1:39015/ using document root <none> and password file <none>
I20260926 02:58:05.093402 8606 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:05.093552 8606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:05.093869 8606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:05.096995 8606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "98c7a502f7684373938ba5318729a587"
format_stamp: "Formatted at 2026-09-26 02:58:05 on dist-test-slave-ktng"
server_key: "530023bcf477dcb7c9f60a5dbd424812"
server_key_iv: "a305c4a47afd95ffd08361e9e0843c81"
server_key_version: "encryptionkey@0"
I20260926 02:58:05.097776 8606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "98c7a502f7684373938ba5318729a587"
format_stamp: "Formatted at 2026-09-26 02:58:05 on dist-test-slave-ktng"
server_key: "530023bcf477dcb7c9f60a5dbd424812"
server_key_iv: "a305c4a47afd95ffd08361e9e0843c81"
server_key_version: "encryptionkey@0"
I20260926 02:58:05.103598 8606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 02:58:05.106979 8621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:05.108438 8606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:58:05.108613 8606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "98c7a502f7684373938ba5318729a587"
format_stamp: "Formatted at 2026-09-26 02:58:05 on dist-test-slave-ktng"
server_key: "530023bcf477dcb7c9f60a5dbd424812"
server_key_iv: "a305c4a47afd95ffd08361e9e0843c81"
server_key_version: "encryptionkey@0"
I20260926 02:58:05.108805 8606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:05.123425 8606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:05.127842 8606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:05.128139 8606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:05.148181 8606 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34633
I20260926 02:58:05.148198 8672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34633 every 8 connection(s)
I20260926 02:58:05.149600 8606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:05.153468 8673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:05.156384 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 8606
I20260926 02:58:05.156566 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:05.157110 20199 external_mini_cluster.cc:1428] Setting key 792a0996de5df69de3dc207797686238
I20260926 02:58:05.168627 8673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587: Bootstrap starting.
I20260926 02:58:05.171947 8673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:05.173013 8673 log.cc:824] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:05.174476 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:05.159210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55453 (local address 127.0.0.1:34633)
0926 02:58:05.160211 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:05.160226 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:05.160273 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:58:05.163679 (+ 3406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:05.163734 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:05.164455 (+ 721us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:05.165005 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:05.165943 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:05.168146 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:05.169672 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:05.170484 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:05.170708 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:05.171974 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:05.171982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:05.171997 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:05.172661 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:05.172751 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:05.172782 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:05.172935 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:05.173168 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 02:58:05.173454 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":469,"threads_started":1}
I20260926 02:58:05.176298 8673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587: No bootstrap required, opened a new log
I20260926 02:58:05.178411 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:05.184873 8673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c7a502f7684373938ba5318729a587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34633 } }
I20260926 02:58:05.185328 8673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:05.185438 8673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c7a502f7684373938ba5318729a587, State: Initialized, Role: FOLLOWER
I20260926 02:58:05.186085 8673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c7a502f7684373938ba5318729a587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34633 } }
I20260926 02:58:05.186318 8673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:05.186503 8673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:05.186668 8673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:05.189630 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:05.182883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47667 (local address 127.0.0.1:34633)
0926 02:58:05.183089 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:05.183102 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:05.183421 (+ 319us) server_negotiation.cc:407] Connection header received
0926 02:58:05.183592 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:05.183598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:05.183679 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:05.183794 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:05.184320 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:05.185362 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:05.186901 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:05.187674 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:05.187816 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:05.188827 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:05.188834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:05.188839 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:05.188860 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:05.188923 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:05.188945 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:05.189044 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:05.189216 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 02:58:05.189312 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:58:05.190050 8673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c7a502f7684373938ba5318729a587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34633 } }
I20260926 02:58:05.190582 8673 leader_election.cc:302] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98c7a502f7684373938ba5318729a587; no voters:
I20260926 02:58:05.191308 8673 leader_election.cc:288] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:05.191433 8683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:05.192123 8683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [term 1 LEADER]: Becoming Leader. State: Replica: 98c7a502f7684373938ba5318729a587, State: Running, Role: LEADER
I20260926 02:58:05.192776 8683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c7a502f7684373938ba5318729a587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34633 } }
I20260926 02:58:05.193692 8673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:05.195673 8684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98c7a502f7684373938ba5318729a587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c7a502f7684373938ba5318729a587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34633 } } }
I20260926 02:58:05.196044 8684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:05.199818 8685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c7a502f7684373938ba5318729a587. Latest consensus state: current_term: 1 leader_uuid: "98c7a502f7684373938ba5318729a587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c7a502f7684373938ba5318729a587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34633 } } }
I20260926 02:58:05.200274 8685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:05.202575 8689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:05.207863 8689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:05.217177 8689 catalog_manager.cc:1347] Generated new cluster ID: 0ccf74fe78c24644b9abb04fac95120e
I20260926 02:58:05.217288 8689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:05.230165 8689 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:05.231559 8689 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:05.241811 8689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98c7a502f7684373938ba5318729a587: Generated new TSK 0
I20260926 02:58:05.242650 8689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:05.252534 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 8606
W20260926 02:58:05.253629 8668 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55453
W20260926 02:58:05.254340 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 8606: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:58:05.254534 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 8606
2026-09-26T02:58:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:05 dist-test-slave-ktng krb5kdc[8704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:05 dist-test-slave-ktng krb5kdc[8704](info): set up 2 sockets
Sep 26 02:58:05 dist-test-slave-ktng krb5kdc[8704](info): commencing operation
krb5kdc: starting...
W20260926 02:58:06.424834 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:06 dist-test-slave-ktng krb5kdc[8704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:58:06.688021 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:07.033391 8717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:07.033692 8717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:07.033900 8717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:07.041182 8717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:07.041306 8717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:07.041402 8717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:07.041505 8717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:07.041586 8717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:07.051684 8717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:07.052986 8717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:07.054288 8717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:07.066573 8725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:07.066628 8723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:07.066654 8722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:07.067638 8717 server_base.cc:1034] running on GCE node
I20260926 02:58:07.068444 8717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:07.069864 8717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:07.071108 8717 hybrid_clock.cc:648] HybridClock initialized: now 1790391487071048 us; error 51 us; skew 500 ppm
I20260926 02:58:07.071548 8717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:07 dist-test-slave-ktng krb5kdc[8704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:07.074949 8717 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:58:07.076622 8717 webserver.cc:460] Webserver started at http://127.0.0.1:35173/ using document root <none> and password file <none>
I20260926 02:58:07.077293 8717 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:07.077437 8717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:07.077767 8717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:07.080782 8717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "b0d76c75a011466d8b112436b4e11ae8"
format_stamp: "Formatted at 2026-09-26 02:58:07 on dist-test-slave-ktng"
server_key: "a893cb8651f0be0912b4e301dabfa817"
server_key_iv: "1ea88b4d47d0a9f835ececd1906e5d62"
server_key_version: "encryptionkey@0"
I20260926 02:58:07.081562 8717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "b0d76c75a011466d8b112436b4e11ae8"
format_stamp: "Formatted at 2026-09-26 02:58:07 on dist-test-slave-ktng"
server_key: "a893cb8651f0be0912b4e301dabfa817"
server_key_iv: "1ea88b4d47d0a9f835ececd1906e5d62"
server_key_version: "encryptionkey@0"
I20260926 02:58:07.086792 8717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 02:58:07.090569 8732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:07.092226 8717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:58:07.092432 8717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "b0d76c75a011466d8b112436b4e11ae8"
format_stamp: "Formatted at 2026-09-26 02:58:07 on dist-test-slave-ktng"
server_key: "a893cb8651f0be0912b4e301dabfa817"
server_key_iv: "1ea88b4d47d0a9f835ececd1906e5d62"
server_key_version: "encryptionkey@0"
I20260926 02:58:07.092646 8717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:07.098601 8717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:07.103060 8717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:07.103443 8717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:07.122885 8717 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46663
I20260926 02:58:07.122946 8783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46663 every 8 connection(s)
I20260926 02:58:07.124413 8717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:07.128409 8784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:07.134598 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 8717
I20260926 02:58:07.134770 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:07.135365 20199 external_mini_cluster.cc:1428] Setting key 82b9e1ac7bda9423389ec92bf095823d
I20260926 02:58:07.139456 8784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8: Bootstrap starting.
I20260926 02:58:07.143049 8784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:07.143508 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:07.137256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50107 (local address 127.0.0.1:46663)
0926 02:58:07.138112 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:07.138123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:07.138164 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:58:07.139971 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:07.139994 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:07.140317 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:07.140622 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:07.141802 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:07.141808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:07.141822 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:07.141836 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:07.141902 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:07.141966 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:07.142132 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:07.142240 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 02:58:07.142441 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":397,"threads_started":1}
I20260926 02:58:07.144275 8784 log.cc:824] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:07.147507 8784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8: No bootstrap required, opened a new log
I20260926 02:58:07.147874 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:07.155812 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:07.153208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60249 (local address 127.0.0.1:46663)
0926 02:58:07.153414 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:07.153422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:07.153833 (+ 411us) server_negotiation.cc:407] Connection header received
0926 02:58:07.153971 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:07.153976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:07.154042 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:07.154160 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:07.155068 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:07.155074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:07.155079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:07.155097 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:07.155147 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:07.155167 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:07.155332 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:07.155437 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 02:58:07.155501 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:58:07.157790 8784 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d76c75a011466d8b112436b4e11ae8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } }
I20260926 02:58:07.158165 8784 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:07.158288 8784 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d76c75a011466d8b112436b4e11ae8, State: Initialized, Role: FOLLOWER
I20260926 02:58:07.158833 8784 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d76c75a011466d8b112436b4e11ae8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } }
I20260926 02:58:07.159091 8784 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:07.159240 8784 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:07.159453 8784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:07.161767 8784 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d76c75a011466d8b112436b4e11ae8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } }
I20260926 02:58:07.162266 8784 leader_election.cc:302] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0d76c75a011466d8b112436b4e11ae8; no voters:
I20260926 02:58:07.165784 8784 leader_election.cc:288] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:07.165876 8794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:07.167520 8794 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [term 1 LEADER]: Becoming Leader. State: Replica: b0d76c75a011466d8b112436b4e11ae8, State: Running, Role: LEADER
I20260926 02:58:07.168159 8794 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d76c75a011466d8b112436b4e11ae8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } }
I20260926 02:58:07.171193 8784 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:07.171355 8796 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0d76c75a011466d8b112436b4e11ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d76c75a011466d8b112436b4e11ae8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } } }
I20260926 02:58:07.171762 8796 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:07.172170 8795 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0d76c75a011466d8b112436b4e11ae8. Latest consensus state: current_term: 1 leader_uuid: "b0d76c75a011466d8b112436b4e11ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d76c75a011466d8b112436b4e11ae8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46663 } } }
I20260926 02:58:07.172545 8795 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:07.181478 8798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:07.186008 8798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:07.195053 8798 catalog_manager.cc:1347] Generated new cluster ID: f7df97bdf37b4daf8bfa5c546a633393
I20260926 02:58:07.195169 8798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:07.208616 8798 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:07.209043 8798 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:07.217905 8798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0d76c75a011466d8b112436b4e11ae8: Generated new TSK 0
I20260926 02:58:07.218700 8798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:07.223591 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 8717
W20260926 02:58:07.224737 8779 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50107
W20260926 02:58:07.225597 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 8717: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:58:07.225795 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 8717
2026-09-26T02:58:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1970 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:07 dist-test-slave-ktng krb5kdc[8816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:07 dist-test-slave-ktng krb5kdc[8816](info): set up 2 sockets
Sep 26 02:58:07 dist-test-slave-ktng krb5kdc[8816](info): commencing operation
krb5kdc: starting...
W20260926 02:58:08.341125 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:08 dist-test-slave-ktng krb5kdc[8816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:58:08.611907 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:08.950510 8829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:08.950817 8829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:08.950914 8829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:08.958559 8829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:08.958698 8829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:08.958747 8829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:08.958802 8829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:08.958854 8829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:08.969089 8829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:08.970345 8829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:08.971673 8829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:08.983559 8834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:08.983471 8837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:08.983520 8835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:08.984778 8829 server_base.cc:1034] running on GCE node
I20260926 02:58:08.985569 8829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:08.987022 8829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:08.988236 8829 hybrid_clock.cc:648] HybridClock initialized: now 1790391488988166 us; error 72 us; skew 500 ppm
I20260926 02:58:08.988626 8829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:08 dist-test-slave-ktng krb5kdc[8816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:08.998131 8829 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:58:08.999814 8829 webserver.cc:460] Webserver started at http://127.0.0.1:42881/ using document root <none> and password file <none>
I20260926 02:58:09.000554 8829 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:09.000710 8829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:09.001031 8829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:09.004180 8829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "b83adb1bb2634f18806b15d2284b7a2b"
format_stamp: "Formatted at 2026-09-26 02:58:08 on dist-test-slave-ktng"
server_key: "9b8dc347848d073077db919939bae59d"
server_key_iv: "3e4c1a276473c87778a635717606dc38"
server_key_version: "encryptionkey@0"
I20260926 02:58:09.004973 8829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "b83adb1bb2634f18806b15d2284b7a2b"
format_stamp: "Formatted at 2026-09-26 02:58:08 on dist-test-slave-ktng"
server_key: "9b8dc347848d073077db919939bae59d"
server_key_iv: "3e4c1a276473c87778a635717606dc38"
server_key_version: "encryptionkey@0"
I20260926 02:58:09.010568 8829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:58:09.014233 8844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:09.015733 8829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:58:09.015920 8829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "b83adb1bb2634f18806b15d2284b7a2b"
format_stamp: "Formatted at 2026-09-26 02:58:08 on dist-test-slave-ktng"
server_key: "9b8dc347848d073077db919939bae59d"
server_key_iv: "3e4c1a276473c87778a635717606dc38"
server_key_version: "encryptionkey@0"
I20260926 02:58:09.016086 8829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:09.031808 8829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:09.036049 8829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:09.036342 8829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:09.056044 8829 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37387
I20260926 02:58:09.056067 8895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37387 every 8 connection(s)
I20260926 02:58:09.057543 8829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:09.058607 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 8829
I20260926 02:58:09.058800 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:09.059367 20199 external_mini_cluster.cc:1428] Setting key b1a7e96daea72d1a5df1bbb31390cfb7
I20260926 02:58:09.062342 8896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:58:09 dist-test-slave-ktng krb5kdc[8816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:58:09.075950 8896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b: Bootstrap starting.
I20260926 02:58:09.080228 8896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:09.081586 8896 log.cc:824] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:09.084619 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:09.061231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:46747 (local address 127.0.0.1:37387)
0926 02:58:09.062019 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:09.062031 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:09.062071 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:58:09.063841 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:09.063865 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:09.064243 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:09.064566 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:09.066361 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:09.067814 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:09.069576 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:09.070230 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:09.070394 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:09.075559 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:09.075570 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:09.075592 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:09.075646 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:09.079699 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:09.080612 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:09.080624 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:09.080633 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:09.080858 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:09.081379 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:09.081386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:09.081389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:09.082069 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:09.082459 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:09.082950 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 02:58:09.083256 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":363,"threads_started":1}
I20260926 02:58:09.085335 8896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b: No bootstrap required, opened a new log
I20260926 02:58:09.088672 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:09.093816 8896 raft_consensus.cc:348] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83adb1bb2634f18806b15d2284b7a2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37387 } }
I20260926 02:58:09.094295 8896 raft_consensus.cc:374] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:09.094426 8896 raft_consensus.cc:729] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83adb1bb2634f18806b15d2284b7a2b, State: Initialized, Role: FOLLOWER
I20260926 02:58:09.095019 8896 consensus_queue.cc:260] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83adb1bb2634f18806b15d2284b7a2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37387 } }
I20260926 02:58:09.095252 8896 raft_consensus.cc:388] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:09.095422 8896 raft_consensus.cc:482] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:09.095592 8896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:09.098454 8896 raft_consensus.cc:504] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83adb1bb2634f18806b15d2284b7a2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37387 } }
I20260926 02:58:09.099076 8896 leader_election.cc:302] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b83adb1bb2634f18806b15d2284b7a2b; no voters:
I20260926 02:58:09.099798 8896 leader_election.cc:288] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:09.099941 8906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:09.101050 8906 raft_consensus.cc:686] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [term 1 LEADER]: Becoming Leader. State: Replica: b83adb1bb2634f18806b15d2284b7a2b, State: Running, Role: LEADER
I20260926 02:58:09.101737 8906 consensus_queue.cc:237] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83adb1bb2634f18806b15d2284b7a2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37387 } }
I20260926 02:58:09.102118 8896 sys_catalog.cc:564] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:09.104592 8908 sys_catalog.cc:455] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b83adb1bb2634f18806b15d2284b7a2b. Latest consensus state: current_term: 1 leader_uuid: "b83adb1bb2634f18806b15d2284b7a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83adb1bb2634f18806b15d2284b7a2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37387 } } }
I20260926 02:58:09.105506 8908 sys_catalog.cc:458] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:09.105194 8907 sys_catalog.cc:455] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b83adb1bb2634f18806b15d2284b7a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83adb1bb2634f18806b15d2284b7a2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37387 } } }
I20260926 02:58:09.106150 8907 sys_catalog.cc:458] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:09.107873 8913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:09.112795 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:09.092960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:57411 (local address 127.0.0.1:37387)
0926 02:58:09.093140 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:09.093147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:09.093665 (+ 518us) server_negotiation.cc:407] Connection header received
0926 02:58:09.093811 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:09.093816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:09.093902 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:09.094011 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:09.096035 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:09.097272 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:09.098606 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:09.099311 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:09.099435 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:09.102696 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:09.102704 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:09.102714 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:09.102752 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:09.107520 (+ 4768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:09.108538 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:09.108547 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:09.108553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:09.108806 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:09.109364 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:09.109371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:09.109377 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:09.109872 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:09.111729 (+ 1857us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:09.111972 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 02:58:09.112226 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:58:09.113205 8913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:09.123478 8913 catalog_manager.cc:1347] Generated new cluster ID: 6e02daa6720d4c54a6851602795211cc
I20260926 02:58:09.123603 8913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:09.143946 8913 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:09.145284 8913 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:09.157567 8913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b83adb1bb2634f18806b15d2284b7a2b: Generated new TSK 0
I20260926 02:58:09.158496 8913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:09.168025 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 8829
I20260926 02:58:09.171383 8891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:58:09.273329 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 8829
W20260926 02:58:09.293761 8814 connection.cc:535] client connection to 127.0.0.1:37387 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37387): Connection reset by peer (error 104)
2026-09-26T02:58:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2050 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:09 dist-test-slave-ktng krb5kdc[8928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:09 dist-test-slave-ktng krb5kdc[8928](info): set up 2 sockets
Sep 26 02:58:09 dist-test-slave-ktng krb5kdc[8928](info): commencing operation
krb5kdc: starting...
W20260926 02:58:10.400720 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:10 dist-test-slave-ktng krb5kdc[8928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:58:10.670711 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:11.012702 8941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:11.013000 8941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:11.013173 8941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:11.020639 8941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:11.020752 8941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:11.020870 8941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:11.020968 8941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:11.021046 8941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:11.030987 8941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:11.032546 8941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:11.034411 8941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:11.047398 8946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:11.047338 8949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:11.047319 8947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:11.048072 8941 server_base.cc:1034] running on GCE node
I20260926 02:58:11.049006 8941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:11.050575 8941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:11.051875 8941 hybrid_clock.cc:648] HybridClock initialized: now 1790391491051778 us; error 58 us; skew 500 ppm
I20260926 02:58:11.052273 8941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:11 dist-test-slave-ktng krb5kdc[8928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:11.055660 8941 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:58:11.057205 8941 webserver.cc:460] Webserver started at http://127.0.0.1:35609/ using document root <none> and password file <none>
I20260926 02:58:11.057953 8941 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:11.058104 8941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:11.058415 8941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:11.061491 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "d3139a43f6204d658ab3a4709755334e"
format_stamp: "Formatted at 2026-09-26 02:58:11 on dist-test-slave-ktng"
server_key: "8fd3d0dbe61121b9959b79f3decac55d"
server_key_iv: "c1942683c2070f5a2aa88b3e50006533"
server_key_version: "encryptionkey@0"
I20260926 02:58:11.062291 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "d3139a43f6204d658ab3a4709755334e"
format_stamp: "Formatted at 2026-09-26 02:58:11 on dist-test-slave-ktng"
server_key: "8fd3d0dbe61121b9959b79f3decac55d"
server_key_iv: "c1942683c2070f5a2aa88b3e50006533"
server_key_version: "encryptionkey@0"
I20260926 02:58:11.067579 8941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:58:11.071017 8956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:11.072520 8941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:58:11.072721 8941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "d3139a43f6204d658ab3a4709755334e"
format_stamp: "Formatted at 2026-09-26 02:58:11 on dist-test-slave-ktng"
server_key: "8fd3d0dbe61121b9959b79f3decac55d"
server_key_iv: "c1942683c2070f5a2aa88b3e50006533"
server_key_version: "encryptionkey@0"
I20260926 02:58:11.072927 8941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:11.090106 8941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:11.094530 8941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:11.094817 8941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:11.113478 8941 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35725
I20260926 02:58:11.113499 9007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35725 every 8 connection(s)
I20260926 02:58:11.114853 8941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:11.117030 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 8941
I20260926 02:58:11.117215 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:11.117794 20199 external_mini_cluster.cc:1428] Setting key a5f9faf1cc3b0b93bfb153d9f4e0ef77
I20260926 02:58:11.119347 9008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:11.130772 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:11.119718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:35491 (local address 127.0.0.1:35725)
0926 02:58:11.120486 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:11.120498 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:11.120538 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:58:11.122142 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:11.122163 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:11.122566 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:11.122990 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:11.123563 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:11.124874 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:11.126241 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:11.126921 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:11.127130 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:11.128457 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:11.128465 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:11.128479 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:11.129052 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:11.129128 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:11.129155 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:11.129339 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:11.129634 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 02:58:11.129810 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":329,"threads_started":1}
I20260926 02:58:11.131072 9008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e: Bootstrap starting.
I20260926 02:58:11.134505 9008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:11.134899 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:11.135668 9008 log.cc:824] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:11.138468 9008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e: No bootstrap required, opened a new log
I20260926 02:58:11.146735 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:11.139610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:60317 (local address 127.0.0.1:35725)
0926 02:58:11.139854 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:11.139863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:11.140264 (+ 401us) server_negotiation.cc:407] Connection header received
0926 02:58:11.140447 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:11.140451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:11.140522 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:11.140614 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:11.141215 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:11.142081 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:11.143936 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:11.144477 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:11.144583 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:11.145905 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:11.145910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:11.145913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:11.145931 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:11.145969 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:11.145983 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:11.146067 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:11.146292 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 02:58:11.146374 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:58:11.148598 9008 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3139a43f6204d658ab3a4709755334e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35725 } }
I20260926 02:58:11.149022 9008 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:11.149134 9008 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3139a43f6204d658ab3a4709755334e, State: Initialized, Role: FOLLOWER
I20260926 02:58:11.149784 9008 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3139a43f6204d658ab3a4709755334e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35725 } }
I20260926 02:58:11.150048 9008 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:11.150219 9008 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:11.150430 9008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:11.152844 9008 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3139a43f6204d658ab3a4709755334e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35725 } }
I20260926 02:58:11.153383 9008 leader_election.cc:302] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3139a43f6204d658ab3a4709755334e; no voters:
I20260926 02:58:11.154176 9008 leader_election.cc:288] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:11.154361 9018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:11.155215 9018 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [term 1 LEADER]: Becoming Leader. State: Replica: d3139a43f6204d658ab3a4709755334e, State: Running, Role: LEADER
I20260926 02:58:11.155861 9018 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3139a43f6204d658ab3a4709755334e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35725 } }
I20260926 02:58:11.156567 9008 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:11.158958 9019 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3139a43f6204d658ab3a4709755334e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3139a43f6204d658ab3a4709755334e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35725 } } }
I20260926 02:58:11.158987 9020 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3139a43f6204d658ab3a4709755334e. Latest consensus state: current_term: 1 leader_uuid: "d3139a43f6204d658ab3a4709755334e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3139a43f6204d658ab3a4709755334e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35725 } } }
I20260926 02:58:11.159489 9019 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:11.159543 9020 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:11.169708 9023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:11.176388 9023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:11.186933 9023 catalog_manager.cc:1347] Generated new cluster ID: 2ddf143d2c294f9ba81a3bd6f1c6e646
I20260926 02:58:11.187057 9023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:11.204020 9023 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:11.205344 9023 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:11.214679 9023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3139a43f6204d658ab3a4709755334e: Generated new TSK 0
I20260926 02:58:11.215821 9023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:11.232517 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 8941
W20260926 02:58:11.234061 9003 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:35491
W20260926 02:58:11.234980 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 8941: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:58:11.235141 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 8941
W20260926 02:58:11.252508 8926 connection.cc:535] client connection to 127.0.0.1:35725 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35725): Connection reset by peer (error 104)
2026-09-26T02:58:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:11 dist-test-slave-ktng krb5kdc[9039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:11 dist-test-slave-ktng krb5kdc[9039](info): set up 2 sockets
Sep 26 02:58:11 dist-test-slave-ktng krb5kdc[9039](info): commencing operation
krb5kdc: starting...
W20260926 02:58:12.388744 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:12 dist-test-slave-ktng krb5kdc[9039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:58:12.667670 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36099
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:13.007426 9052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:13.007733 9052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:13.007959 9052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:13.015374 9052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:13.015483 9052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:13.015604 9052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:13.015717 9052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:13.015811 9052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:13.025902 9052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:13.027276 9052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:13.028627 9052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:13.040823 9060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:13.040881 9058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:13.040904 9057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:13.041671 9052 server_base.cc:1034] running on GCE node
I20260926 02:58:13.042480 9052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:13.044046 9052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:13.045255 9052 hybrid_clock.cc:648] HybridClock initialized: now 1790391493045222 us; error 40 us; skew 500 ppm
I20260926 02:58:13.045688 9052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:13 dist-test-slave-ktng krb5kdc[9039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:13.049212 9052 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:58:13.050748 9052 webserver.cc:460] Webserver started at http://127.0.0.1:38641/ using document root <none> and password file <none>
I20260926 02:58:13.051570 9052 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:13.051715 9052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:13.052033 9052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:13.055193 9052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "041e1504eb1e473281eb8d8bb64f50c5"
format_stamp: "Formatted at 2026-09-26 02:58:13 on dist-test-slave-ktng"
server_key: "8fedd2e59ed223b79863ad35e551060e"
server_key_iv: "a0624bbe50d824cd5842c79a12c33e3b"
server_key_version: "encryptionkey@0"
I20260926 02:58:13.056052 9052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "041e1504eb1e473281eb8d8bb64f50c5"
format_stamp: "Formatted at 2026-09-26 02:58:13 on dist-test-slave-ktng"
server_key: "8fedd2e59ed223b79863ad35e551060e"
server_key_iv: "a0624bbe50d824cd5842c79a12c33e3b"
server_key_version: "encryptionkey@0"
I20260926 02:58:13.061537 9052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:58:13.065277 9067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:13.066741 9052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:58:13.066931 9052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "041e1504eb1e473281eb8d8bb64f50c5"
format_stamp: "Formatted at 2026-09-26 02:58:13 on dist-test-slave-ktng"
server_key: "8fedd2e59ed223b79863ad35e551060e"
server_key_iv: "a0624bbe50d824cd5842c79a12c33e3b"
server_key_version: "encryptionkey@0"
I20260926 02:58:13.067147 9052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:13.073040 9052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:13.077461 9052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:13.077788 9052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:13.096860 9052 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36099
I20260926 02:58:13.096905 9118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36099 every 8 connection(s)
I20260926 02:58:13.098295 9052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:13.102056 9119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:13.102542 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 9052
I20260926 02:58:13.102721 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:13.103271 20199 external_mini_cluster.cc:1428] Setting key a5c7f8cfb4f8099db249871fcf7b2c24
I20260926 02:58:13.111694 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:13.105129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:59521 (local address 127.0.0.1:36099)
0926 02:58:13.105960 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:13.105972 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:13.106010 (+ 38us) server_negotiation.cc:407] Connection header received
0926 02:58:13.108286 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:13.108311 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:13.108649 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:13.108994 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:13.110069 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:13.110076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:13.110088 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:13.110102 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:13.110157 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:13.110180 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:13.110341 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:13.110437 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 02:58:13.110638 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":387,"threads_started":1}
I20260926 02:58:13.115545 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:13.116042 9119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5: Bootstrap starting.
I20260926 02:58:13.119184 9119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:13.120553 9119 log.cc:824] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:13.122769 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:13.120212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:46457 (local address 127.0.0.1:36099)
0926 02:58:13.120416 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:13.120422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:13.120786 (+ 364us) server_negotiation.cc:407] Connection header received
0926 02:58:13.120983 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:13.120987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:13.121034 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:13.121147 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:13.122125 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:13.122135 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:13.122138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:13.122152 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:13.122189 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:13.122203 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:13.122322 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:13.122449 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 02:58:13.122512 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:58:13.123792 9119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5: No bootstrap required, opened a new log
I20260926 02:58:13.131383 9119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041e1504eb1e473281eb8d8bb64f50c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36099 } }
I20260926 02:58:13.131824 9119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:13.131937 9119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041e1504eb1e473281eb8d8bb64f50c5, State: Initialized, Role: FOLLOWER
I20260926 02:58:13.132658 9119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041e1504eb1e473281eb8d8bb64f50c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36099 } }
I20260926 02:58:13.132874 9119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:13.132983 9119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:13.133104 9119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:13.135358 9119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041e1504eb1e473281eb8d8bb64f50c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36099 } }
I20260926 02:58:13.135828 9119 leader_election.cc:302] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 041e1504eb1e473281eb8d8bb64f50c5; no voters:
I20260926 02:58:13.136348 9119 leader_election.cc:288] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:13.136505 9129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:13.137750 9129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [term 1 LEADER]: Becoming Leader. State: Replica: 041e1504eb1e473281eb8d8bb64f50c5, State: Running, Role: LEADER
I20260926 02:58:13.138342 9129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041e1504eb1e473281eb8d8bb64f50c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36099 } }
I20260926 02:58:13.138728 9119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:13.141286 9131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 041e1504eb1e473281eb8d8bb64f50c5. Latest consensus state: current_term: 1 leader_uuid: "041e1504eb1e473281eb8d8bb64f50c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041e1504eb1e473281eb8d8bb64f50c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36099 } } }
I20260926 02:58:13.141656 9131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:13.142140 9130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "041e1504eb1e473281eb8d8bb64f50c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041e1504eb1e473281eb8d8bb64f50c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36099 } } }
I20260926 02:58:13.142427 9130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:13.143720 9136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:13.150194 9136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:13.160187 9136 catalog_manager.cc:1347] Generated new cluster ID: c3869da1aec54f318f07c60a752b8e41
I20260926 02:58:13.160351 9136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:13.190872 9136 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:13.191376 9136 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:13.198828 9136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 041e1504eb1e473281eb8d8bb64f50c5: Generated new TSK 0
I20260926 02:58:13.199761 9136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:13.213063 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9052
W20260926 02:58:13.214223 9114 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:59521
W20260926 02:58:13.215071 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9052: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:58:13.215225 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 9052
2026-09-26T02:58:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:13 dist-test-slave-ktng krb5kdc[9150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:13 dist-test-slave-ktng krb5kdc[9150](info): set up 2 sockets
Sep 26 02:58:13 dist-test-slave-ktng krb5kdc[9150](info): commencing operation
krb5kdc: starting...
W20260926 02:58:14.340914 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:14 dist-test-slave-ktng krb5kdc[9150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:58:14.608489 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:14.946908 9163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:14.947266 9163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:14.947474 9163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:14.954737 9163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:14.954890 9163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:14.954999 9163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:14.955112 9163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:14.955199 9163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:14.965159 9163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:14.966490 9163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:14.967882 9163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:14.981547 9168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:14.981647 9169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:14.981603 9171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:14.982748 9163 server_base.cc:1034] running on GCE node
I20260926 02:58:14.983747 9163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:14.985361 9163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:14.986588 9163 hybrid_clock.cc:648] HybridClock initialized: now 1790391494986545 us; error 31 us; skew 500 ppm
I20260926 02:58:14.987030 9163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:14 dist-test-slave-ktng krb5kdc[9150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:14.997160 9163 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:58:14.998974 9163 webserver.cc:460] Webserver started at http://127.0.0.1:41423/ using document root <none> and password file <none>
I20260926 02:58:14.999814 9163 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:14.999966 9163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:15.000317 9163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:15.003814 9163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "f26a247ed8d34b6aac79fa5552d7c560"
format_stamp: "Formatted at 2026-09-26 02:58:14 on dist-test-slave-ktng"
server_key: "5c0433210f02366128a75ded6d13c4f2"
server_key_iv: "f385039323a2654996bf63d01cea0167"
server_key_version: "encryptionkey@0"
I20260926 02:58:15.004725 9163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "f26a247ed8d34b6aac79fa5552d7c560"
format_stamp: "Formatted at 2026-09-26 02:58:14 on dist-test-slave-ktng"
server_key: "5c0433210f02366128a75ded6d13c4f2"
server_key_iv: "f385039323a2654996bf63d01cea0167"
server_key_version: "encryptionkey@0"
I20260926 02:58:15.010447 9163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:58:15.014032 9178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:15.015390 9163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:58:15.015588 9163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "f26a247ed8d34b6aac79fa5552d7c560"
format_stamp: "Formatted at 2026-09-26 02:58:14 on dist-test-slave-ktng"
server_key: "5c0433210f02366128a75ded6d13c4f2"
server_key_iv: "f385039323a2654996bf63d01cea0167"
server_key_version: "encryptionkey@0"
I20260926 02:58:15.015801 9163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:15.031631 9163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:15.035991 9163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:15.036275 9163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:15.055225 9163 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45115
I20260926 02:58:15.055248 9229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45115 every 8 connection(s)
I20260926 02:58:15.056670 9163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:15.060540 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:15.064044 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 9163
I20260926 02:58:15.064224 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:15.064786 20199 external_mini_cluster.cc:1428] Setting key 762e190b25281c4b028d77c74739eed8
I20260926 02:58:15.072788 9230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560: Bootstrap starting.
I20260926 02:58:15.076046 9230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:15.077423 9230 log.cc:824] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:58:15 dist-test-slave-ktng krb5kdc[9150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:58:15.080554 9230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560: No bootstrap required, opened a new log
I20260926 02:58:15.088025 9233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:15.066683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:58077 (local address 127.0.0.1:45115)
0926 02:58:15.067509 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:15.067521 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:15.067564 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:58:15.069273 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:15.069298 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:15.069747 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:15.070157 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:15.071785 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:15.073178 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:15.074476 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:15.075170 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:15.075388 (+ 218us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:15.080273 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:15.080281 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:15.080296 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:15.080331 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:15.083663 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:15.084521 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:15.084527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:15.084533 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:15.084702 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:15.085205 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:15.085212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:15.085214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:15.085745 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:15.086099 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:15.086618 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 02:58:15.086941 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":325,"threads_started":1}
I20260926 02:58:15.088862 9230 raft_consensus.cc:348] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26a247ed8d34b6aac79fa5552d7c560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45115 } }
I20260926 02:58:15.089262 9230 raft_consensus.cc:374] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:15.089396 9230 raft_consensus.cc:729] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26a247ed8d34b6aac79fa5552d7c560, State: Initialized, Role: FOLLOWER
I20260926 02:58:15.090031 9230 consensus_queue.cc:260] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26a247ed8d34b6aac79fa5552d7c560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45115 } }
I20260926 02:58:15.090305 9230 raft_consensus.cc:388] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:15.090478 9230 raft_consensus.cc:482] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:15.090724 9230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:15.092049 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:15.093302 9230 raft_consensus.cc:504] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26a247ed8d34b6aac79fa5552d7c560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45115 } }
I20260926 02:58:15.093796 9230 leader_election.cc:302] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f26a247ed8d34b6aac79fa5552d7c560; no voters:
I20260926 02:58:15.094416 9230 leader_election.cc:288] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:15.094594 9235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:15.096225 9235 raft_consensus.cc:686] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [term 1 LEADER]: Becoming Leader. State: Replica: f26a247ed8d34b6aac79fa5552d7c560, State: Running, Role: LEADER
I20260926 02:58:15.097002 9235 consensus_queue.cc:237] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26a247ed8d34b6aac79fa5552d7c560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45115 } }
I20260926 02:58:15.097195 9230 sys_catalog.cc:564] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:15.100019 9239 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f26a247ed8d34b6aac79fa5552d7c560. Latest consensus state: current_term: 1 leader_uuid: "f26a247ed8d34b6aac79fa5552d7c560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26a247ed8d34b6aac79fa5552d7c560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45115 } } }
I20260926 02:58:15.100402 9239 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:15.100952 9237 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f26a247ed8d34b6aac79fa5552d7c560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26a247ed8d34b6aac79fa5552d7c560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45115 } } }
I20260926 02:58:15.101305 9237 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:15.102427 9245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:15.111332 9245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:15.125980 9233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:15.098418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:59971 (local address 127.0.0.1:45115)
0926 02:58:15.098566 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:15.098570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:15.103190 (+ 4620us) server_negotiation.cc:407] Connection header received
0926 02:58:15.103478 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:15.103484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:15.103580 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:15.103711 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:15.106143 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:15.107463 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:15.109436 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:15.110136 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:15.110246 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:15.116683 (+ 6437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:15.116692 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:15.116697 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:15.116741 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:15.121609 (+ 4868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:15.122887 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:15.122909 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:15.122915 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:15.123099 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:15.123868 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:15.123876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:15.123882 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:15.124450 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:15.124716 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:15.125476 (+ 760us) server_negotiation.cc:299] Negotiation successful
0926 02:58:15.125670 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 02:58:15.129163 9245 catalog_manager.cc:1347] Generated new cluster ID: fdf02c9648f54ce989952f86eb0c7a02
I20260926 02:58:15.129345 9245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:15.150525 9245 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:15.152053 9245 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:15.162832 9245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f26a247ed8d34b6aac79fa5552d7c560: Generated new TSK 0
I20260926 02:58:15.164175 9245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:15.177930 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9163
I20260926 02:58:15.179124 9225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:58:15.274313 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 9163
W20260926 02:58:15.292071 9148 connection.cc:535] client connection to 127.0.0.1:45115 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45115): Connection reset by peer (error 104)
2026-09-26T02:58:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2065 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:15 dist-test-slave-ktng krb5kdc[9263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:15 dist-test-slave-ktng krb5kdc[9263](info): set up 2 sockets
Sep 26 02:58:15 dist-test-slave-ktng krb5kdc[9263](info): commencing operation
krb5kdc: starting...
W20260926 02:58:16.452809 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:16 dist-test-slave-ktng krb5kdc[9263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:58:16.719105 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:17.053593 9276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:17.053972 9276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:17.054149 9276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:17.061504 9276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:17.061684 9276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:17.061808 9276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:17.061924 9276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:17.062013 9276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:17.071938 9276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:17.073307 9276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:17.074662 9276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:17.087088 9281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:17.087092 9284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:17.087030 9282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:17.087787 9276 server_base.cc:1034] running on GCE node
I20260926 02:58:17.088601 9276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:17.090065 9276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:17.091277 9276 hybrid_clock.cc:648] HybridClock initialized: now 1790391497091232 us; error 41 us; skew 500 ppm
I20260926 02:58:17.091706 9276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:17 dist-test-slave-ktng krb5kdc[9263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:17.094951 9276 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:58:17.096519 9276 webserver.cc:460] Webserver started at http://127.0.0.1:44523/ using document root <none> and password file <none>
I20260926 02:58:17.097216 9276 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:17.097349 9276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:17.097687 9276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:17.100759 9276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "f36eb8b651534193ac9426b4fca949ab"
format_stamp: "Formatted at 2026-09-26 02:58:17 on dist-test-slave-ktng"
server_key: "d6572682dc5484c82d167c826e9382e3"
server_key_iv: "79a4a918fd60b2564f38ea061aaa464d"
server_key_version: "encryptionkey@0"
I20260926 02:58:17.101536 9276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "f36eb8b651534193ac9426b4fca949ab"
format_stamp: "Formatted at 2026-09-26 02:58:17 on dist-test-slave-ktng"
server_key: "d6572682dc5484c82d167c826e9382e3"
server_key_iv: "79a4a918fd60b2564f38ea061aaa464d"
server_key_version: "encryptionkey@0"
I20260926 02:58:17.106679 9276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:58:17.110270 9291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:17.111737 9276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:58:17.111956 9276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "f36eb8b651534193ac9426b4fca949ab"
format_stamp: "Formatted at 2026-09-26 02:58:17 on dist-test-slave-ktng"
server_key: "d6572682dc5484c82d167c826e9382e3"
server_key_iv: "79a4a918fd60b2564f38ea061aaa464d"
server_key_version: "encryptionkey@0"
I20260926 02:58:17.112154 9276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:17.153937 9276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:17.158607 9276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:17.158928 9276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:17.180840 9276 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33453
I20260926 02:58:17.180842 9342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33453 every 8 connection(s)
I20260926 02:58:17.182265 9276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:17.184598 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 9276
I20260926 02:58:17.184768 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:17.185190 20199 external_mini_cluster.cc:1428] Setting key fc7d0ca8f67eaee2073c56a844b9a8c9
I20260926 02:58:17.186425 9343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:17.197824 9346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:17.186974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:58407 (local address 127.0.0.1:33453)
0926 02:58:17.187751 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:17.187763 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:17.187801 (+ 38us) server_negotiation.cc:407] Connection header received
0926 02:58:17.189434 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:17.189454 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:17.189851 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:17.190231 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:17.190765 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:17.192145 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:17.193438 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:17.194093 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:17.194298 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:17.195607 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:17.195615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:17.195631 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:17.196195 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:17.196269 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:17.196296 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:17.196468 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:17.196729 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 02:58:17.196899 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":379,"threads_started":1}
I20260926 02:58:17.198553 9343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab: Bootstrap starting.
I20260926 02:58:17.201823 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:17.202152 9343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:17.203161 9343 log.cc:824] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:17.206146 9343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab: No bootstrap required, opened a new log
I20260926 02:58:17.213406 9346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:17.206346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:44715 (local address 127.0.0.1:33453)
0926 02:58:17.206502 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:17.206506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:17.207178 (+ 672us) server_negotiation.cc:407] Connection header received
0926 02:58:17.207435 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:17.207439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:17.207497 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:17.207580 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:17.208222 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:17.209023 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:17.210872 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:17.211381 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:17.211478 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:17.212633 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:17.212638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:17.212641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:17.212658 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:17.212695 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:17.212709 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:17.212796 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:17.213002 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 02:58:17.213046 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 02:58:17.215960 9343 raft_consensus.cc:348] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36eb8b651534193ac9426b4fca949ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33453 } }
I20260926 02:58:17.216337 9343 raft_consensus.cc:374] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:17.216435 9343 raft_consensus.cc:729] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36eb8b651534193ac9426b4fca949ab, State: Initialized, Role: FOLLOWER
I20260926 02:58:17.217007 9343 consensus_queue.cc:260] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36eb8b651534193ac9426b4fca949ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33453 } }
I20260926 02:58:17.217235 9343 raft_consensus.cc:388] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:17.217381 9343 raft_consensus.cc:482] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:17.217576 9343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:17.219870 9343 raft_consensus.cc:504] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36eb8b651534193ac9426b4fca949ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33453 } }
I20260926 02:58:17.220374 9343 leader_election.cc:302] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f36eb8b651534193ac9426b4fca949ab; no voters:
I20260926 02:58:17.221081 9343 leader_election.cc:288] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:17.221176 9353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:17.221956 9353 raft_consensus.cc:686] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [term 1 LEADER]: Becoming Leader. State: Replica: f36eb8b651534193ac9426b4fca949ab, State: Running, Role: LEADER
I20260926 02:58:17.222536 9353 consensus_queue.cc:237] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36eb8b651534193ac9426b4fca949ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33453 } }
I20260926 02:58:17.223271 9343 sys_catalog.cc:564] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:17.225194 9354 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f36eb8b651534193ac9426b4fca949ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36eb8b651534193ac9426b4fca949ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33453 } } }
I20260926 02:58:17.225565 9354 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:17.226011 9355 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader f36eb8b651534193ac9426b4fca949ab. Latest consensus state: current_term: 1 leader_uuid: "f36eb8b651534193ac9426b4fca949ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36eb8b651534193ac9426b4fca949ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33453 } } }
I20260926 02:58:17.226438 9355 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:17.227620 9358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:17.232975 9358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:17.245824 9358 catalog_manager.cc:1347] Generated new cluster ID: 0433f66b3d534fd3ace9abe82b3f073d
I20260926 02:58:17.245955 9358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:17.260533 9358 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:17.262342 9358 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:17.268118 9358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f36eb8b651534193ac9426b4fca949ab: Generated new TSK 0
I20260926 02:58:17.268918 9358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:17.277711 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9276
W20260926 02:58:17.278745 9338 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:58407
W20260926 02:58:17.279492 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9276: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:58:17.279644 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 9276
W20260926 02:58:17.297446 9261 connection.cc:535] client connection to 127.0.0.1:33453 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33453): Connection reset by peer (error 104)
2026-09-26T02:58:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2002 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:17 dist-test-slave-ktng krb5kdc[9374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:17 dist-test-slave-ktng krb5kdc[9374](info): set up 2 sockets
Sep 26 02:58:17 dist-test-slave-ktng krb5kdc[9374](info): commencing operation
krb5kdc: starting...
W20260926 02:58:18.392691 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:18 dist-test-slave-ktng krb5kdc[9374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:58:18.655275 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:18.994414 9387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:18.994730 9387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:18.994832 9387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:19.002451 9387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:19.002585 9387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:19.002684 9387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:19.002887 9387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:19.002955 9387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:19.013361 9387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46231
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:19.014773 9387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:19.016220 9387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:19.028491 9392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:19.028393 9393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:19.028429 9395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:19.029152 9387 server_base.cc:1034] running on GCE node
I20260926 02:58:19.029893 9387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:19.031370 9387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:19.032594 9387 hybrid_clock.cc:648] HybridClock initialized: now 1790391499032550 us; error 35 us; skew 500 ppm
I20260926 02:58:19.033001 9387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:19 dist-test-slave-ktng krb5kdc[9374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:19.036339 9387 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:58:19.037813 9387 webserver.cc:460] Webserver started at http://127.0.0.1:33535/ using document root <none> and password file <none>
I20260926 02:58:19.038532 9387 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:19.038662 9387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:19.038976 9387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:19.042114 9387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "6ed5b94adff9489fa279010f38f69689"
format_stamp: "Formatted at 2026-09-26 02:58:19 on dist-test-slave-ktng"
server_key: "6ed18198729d0d711e9fd887466febbb"
server_key_iv: "cdd61b8ee4d9ad5090b2f11786e50bb1"
server_key_version: "encryptionkey@0"
I20260926 02:58:19.042879 9387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "6ed5b94adff9489fa279010f38f69689"
format_stamp: "Formatted at 2026-09-26 02:58:19 on dist-test-slave-ktng"
server_key: "6ed18198729d0d711e9fd887466febbb"
server_key_iv: "cdd61b8ee4d9ad5090b2f11786e50bb1"
server_key_version: "encryptionkey@0"
I20260926 02:58:19.048450 9387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 02:58:19.052320 9402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:19.053957 9387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:58:19.054172 9387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "6ed5b94adff9489fa279010f38f69689"
format_stamp: "Formatted at 2026-09-26 02:58:19 on dist-test-slave-ktng"
server_key: "6ed18198729d0d711e9fd887466febbb"
server_key_iv: "cdd61b8ee4d9ad5090b2f11786e50bb1"
server_key_version: "encryptionkey@0"
I20260926 02:58:19.054380 9387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:19.060595 9387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:19.065212 9387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:19.065542 9387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:19.084036 9387 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46231
I20260926 02:58:19.084074 9453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46231 every 8 connection(s)
I20260926 02:58:19.085433 9387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:19.089190 9454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:19.091472 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 9387
I20260926 02:58:19.091650 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:19.092186 20199 external_mini_cluster.cc:1428] Setting key 44fbabb258b7275b34b5f2ad6c45c191
I20260926 02:58:19.099937 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:19.093914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:52795 (local address 127.0.0.1:46231)
0926 02:58:19.094728 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:19.094740 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:19.094786 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:58:19.096465 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:19.096490 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:19.096842 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:19.097154 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:19.098279 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:19.098286 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:19.098298 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:19.098336 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:19.098400 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:19.098419 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:19.098600 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:19.098699 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 02:58:19.098919 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":384,"threads_started":1}
I20260926 02:58:19.101059 9454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689: Bootstrap starting.
I20260926 02:58:19.103955 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:19.104851 9454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:19.106050 9454 log.cc:824] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:19.108923 9454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689: No bootstrap required, opened a new log
I20260926 02:58:19.111776 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:19.109092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:36513 (local address 127.0.0.1:46231)
0926 02:58:19.109276 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:19.109285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:19.109829 (+ 544us) server_negotiation.cc:407] Connection header received
0926 02:58:19.109993 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:19.109997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:19.110041 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:19.110118 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:19.111078 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:19.111082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:19.111085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:19.111100 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:19.111142 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:19.111156 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:19.111246 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:19.111415 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 02:58:19.111469 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:58:19.117287 9454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed5b94adff9489fa279010f38f69689" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46231 } }
I20260926 02:58:19.117690 9454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:19.117810 9454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed5b94adff9489fa279010f38f69689, State: Initialized, Role: FOLLOWER
I20260926 02:58:19.118379 9454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed5b94adff9489fa279010f38f69689" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46231 } }
I20260926 02:58:19.118634 9454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:19.118790 9454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:19.118968 9454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:19.121225 9454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed5b94adff9489fa279010f38f69689" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46231 } }
I20260926 02:58:19.121721 9454 leader_election.cc:302] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ed5b94adff9489fa279010f38f69689; no voters:
I20260926 02:58:19.122334 9454 leader_election.cc:288] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:19.122481 9464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:19.123230 9464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [term 1 LEADER]: Becoming Leader. State: Replica: 6ed5b94adff9489fa279010f38f69689, State: Running, Role: LEADER
I20260926 02:58:19.123853 9464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed5b94adff9489fa279010f38f69689" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46231 } }
I20260926 02:58:19.124509 9454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:19.126770 9465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ed5b94adff9489fa279010f38f69689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed5b94adff9489fa279010f38f69689" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46231 } } }
I20260926 02:58:19.127280 9465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:19.127862 9466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ed5b94adff9489fa279010f38f69689. Latest consensus state: current_term: 1 leader_uuid: "6ed5b94adff9489fa279010f38f69689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed5b94adff9489fa279010f38f69689" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46231 } } }
I20260926 02:58:19.128191 9466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:19.132115 9471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:19.137038 9471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:19.147511 9471 catalog_manager.cc:1347] Generated new cluster ID: d44df86a0cdd4407ba6f6920cbb16fff
I20260926 02:58:19.147655 9471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:19.177053 9471 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:19.177559 9471 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:19.187842 9471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ed5b94adff9489fa279010f38f69689: Generated new TSK 0
I20260926 02:58:19.188875 9471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:19.204903 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9387
W20260926 02:58:19.206307 9449 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.84:52795
W20260926 02:58:19.207100 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9387: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:58:19.207271 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 9387
2026-09-26T02:58:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1927 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:19 dist-test-slave-ktng krb5kdc[9485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:19 dist-test-slave-ktng krb5kdc[9485](info): set up 2 sockets
Sep 26 02:58:19 dist-test-slave-ktng krb5kdc[9485](info): commencing operation
krb5kdc: starting...
W20260926 02:58:20.324743 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:20 dist-test-slave-ktng krb5kdc[9485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:58:20.587471 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:36179
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:36179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:20.927187 9498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:20.927554 9498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:20.927651 9498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:20.934965 9498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:20.935081 9498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:20.935127 9498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:20.935185 9498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:20.935230 9498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:20.945205 9498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:36179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:36179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:20.946462 9498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:20.947846 9498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:20.960815 9504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:20.960872 9503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:20.960773 9506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:20.965145 9498 server_base.cc:1034] running on GCE node
I20260926 02:58:20.966118 9498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:20.967738 9498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:20.968983 9498 hybrid_clock.cc:648] HybridClock initialized: now 1790391500968945 us; error 40 us; skew 500 ppm
I20260926 02:58:20.969362 9498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:20 dist-test-slave-ktng krb5kdc[9485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391500, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:20.977141 9498 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:58:20.978875 9498 webserver.cc:460] Webserver started at http://127.19.185.254:37809/ using document root <none> and password file <none>
I20260926 02:58:20.979691 9498 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:20.979802 9498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:20.980068 9498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:20.983228 9498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "4eddd0738c404410956403b5e497a964"
format_stamp: "Formatted at 2026-09-26 02:58:20 on dist-test-slave-ktng"
server_key: "c41268c9bc1e793bed1cc981330655d7"
server_key_iv: "03ce63f05603c27b5949ce316d827247"
server_key_version: "encryptionkey@0"
I20260926 02:58:20.984042 9498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "4eddd0738c404410956403b5e497a964"
format_stamp: "Formatted at 2026-09-26 02:58:20 on dist-test-slave-ktng"
server_key: "c41268c9bc1e793bed1cc981330655d7"
server_key_iv: "03ce63f05603c27b5949ce316d827247"
server_key_version: "encryptionkey@0"
I20260926 02:58:20.989898 9498 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:58:20.993909 9513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:20.995643 9498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:58:20.995802 9498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "4eddd0738c404410956403b5e497a964"
format_stamp: "Formatted at 2026-09-26 02:58:20 on dist-test-slave-ktng"
server_key: "c41268c9bc1e793bed1cc981330655d7"
server_key_iv: "03ce63f05603c27b5949ce316d827247"
server_key_version: "encryptionkey@0"
I20260926 02:58:20.996039 9498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:21.019547 9498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:21.024094 9498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:21.024394 9498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:21.043812 9498 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:36179
I20260926 02:58:21.043831 9564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:36179 every 8 connection(s)
I20260926 02:58:21.045241 9498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:21.049199 9565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:21.054142 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 9498
I20260926 02:58:21.054373 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:21.054821 20199 external_mini_cluster.cc:1428] Setting key ee3842e396345311c736e3ab192c7ffd
I20260926 02:58:21.060623 9565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964: Bootstrap starting.
I20260926 02:58:21.064563 9565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:21.065820 9565 log.cc:824] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:21.068882 9565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964: No bootstrap required, opened a new log
Sep 26 02:58:21 dist-test-slave-ktng krb5kdc[9485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:58:21.077127 9565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eddd0738c404410956403b5e497a964" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36179 } }
I20260926 02:58:21.077608 9565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:21.077725 9565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eddd0738c404410956403b5e497a964, State: Initialized, Role: FOLLOWER
I20260926 02:58:21.078315 9565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eddd0738c404410956403b5e497a964" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36179 } }
I20260926 02:58:21.078567 9565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:21.078606 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:21.056694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34005 (local address 127.19.185.254:36179)
0926 02:58:21.057606 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:21.057618 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:21.057678 (+ 60us) server_negotiation.cc:407] Connection header received
0926 02:58:21.059481 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:21.059505 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:21.059904 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:21.060230 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:21.062166 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:21.063589 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:21.065397 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:21.066146 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:21.066315 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:21.070834 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:21.070845 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:21.070860 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:21.070908 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:21.074127 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:21.074923 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:21.074930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:21.074936 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:21.075102 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:21.075644 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:21.075648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:21.075651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:21.076202 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:21.076538 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:21.077020 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 02:58:21.077320 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":429,"threads_started":1}
I20260926 02:58:21.078715 9565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:21.078958 9565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:21.081950 9565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eddd0738c404410956403b5e497a964" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36179 } }
I20260926 02:58:21.082475 9565 leader_election.cc:302] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4eddd0738c404410956403b5e497a964; no voters:
I20260926 02:58:21.082943 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:21.083565 9565 leader_election.cc:288] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:21.083699 9570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:21.084867 9570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [term 1 LEADER]: Becoming Leader. State: Replica: 4eddd0738c404410956403b5e497a964, State: Running, Role: LEADER
I20260926 02:58:21.085664 9565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:21.085539 9570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eddd0738c404410956403b5e497a964" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36179 } }
I20260926 02:58:21.092064 9571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4eddd0738c404410956403b5e497a964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eddd0738c404410956403b5e497a964" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36179 } } }
I20260926 02:58:21.092238 9572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4eddd0738c404410956403b5e497a964. Latest consensus state: current_term: 1 leader_uuid: "4eddd0738c404410956403b5e497a964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eddd0738c404410956403b5e497a964" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36179 } } }
I20260926 02:58:21.092793 9572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:21.092836 9571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:21.095281 9580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:21.111598 9580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:21.132025 9580 catalog_manager.cc:1347] Generated new cluster ID: a1de5087b76c4e229dd2838cb632013c
I20260926 02:58:21.132174 9580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:21.133736 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:21.105027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38021 (local address 127.19.185.254:36179)
0926 02:58:21.105296 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:21.105307 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:21.112515 (+ 7208us) server_negotiation.cc:407] Connection header received
0926 02:58:21.112864 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:21.112869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:21.112949 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:21.113043 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:21.115820 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:21.117416 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:21.119642 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:21.120415 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:21.120592 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:21.124478 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:21.124486 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:21.124492 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:21.124538 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:21.128817 (+ 4279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:21.129909 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:21.129916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:21.129920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:21.130079 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:21.130736 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:21.130742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:21.130746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:21.131269 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:21.131591 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:21.133094 (+ 1503us) server_negotiation.cc:299] Negotiation successful
0926 02:58:21.133306 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:58:21.173794 9580 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:21.175191 9580 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:21.181746 9580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4eddd0738c404410956403b5e497a964: Generated new TSK 0
I20260926 02:58:21.182560 9580 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:21.197273 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9498
I20260926 02:58:21.198912 9560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:58:21.293942 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 9498
W20260926 02:58:21.311501 9483 connection.cc:535] client connection to 127.19.185.254:36179 recv error: Network error: failed to read from TLS socket (remote: 127.19.185.254:36179): Connection reset by peer (error 104)
2026-09-26T02:58:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2086 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:21 dist-test-slave-ktng krb5kdc[9597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:21 dist-test-slave-ktng krb5kdc[9597](info): set up 2 sockets
Sep 26 02:58:21 dist-test-slave-ktng krb5kdc[9597](info): commencing operation
krb5kdc: starting...
W20260926 02:58:22.412883 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:22 dist-test-slave-ktng krb5kdc[9597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:58:22.674218 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:33987
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:37319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:33987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:23.004006 9610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:23.004297 9610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:23.004455 9610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:23.011819 9610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:23.011932 9610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:23.012048 9610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:23.012151 9610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:23.012228 9610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:23.022159 9610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:37319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:33987
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:33987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:23.023499 9610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:23.024878 9610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:23.037354 9618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:23.037544 9616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:23.037642 9615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:23.038182 9610 server_base.cc:1034] running on GCE node
I20260926 02:58:23.039003 9610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:23.040501 9610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:23.041741 9610 hybrid_clock.cc:648] HybridClock initialized: now 1790391503041688 us; error 44 us; skew 500 ppm
I20260926 02:58:23.042178 9610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:23 dist-test-slave-ktng krb5kdc[9597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391503, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:23.045483 9610 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:58:23.047060 9610 webserver.cc:460] Webserver started at http://127.19.185.254:39669/ using document root <none> and password file <none>
I20260926 02:58:23.047827 9610 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:23.047972 9610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:23.048290 9610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:23.051412 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "f469637a92f2448da318819e7ee58bb0"
format_stamp: "Formatted at 2026-09-26 02:58:23 on dist-test-slave-ktng"
server_key: "2401503590e22976c451bcede90198bb"
server_key_iv: "0961a465cb41f663d493453d1f95512c"
server_key_version: "encryptionkey@0"
I20260926 02:58:23.052335 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "f469637a92f2448da318819e7ee58bb0"
format_stamp: "Formatted at 2026-09-26 02:58:23 on dist-test-slave-ktng"
server_key: "2401503590e22976c451bcede90198bb"
server_key_iv: "0961a465cb41f663d493453d1f95512c"
server_key_version: "encryptionkey@0"
I20260926 02:58:23.057931 9610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:58:23.061826 9625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:23.063280 9610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:58:23.063548 9610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "f469637a92f2448da318819e7ee58bb0"
format_stamp: "Formatted at 2026-09-26 02:58:23 on dist-test-slave-ktng"
server_key: "2401503590e22976c451bcede90198bb"
server_key_iv: "0961a465cb41f663d493453d1f95512c"
server_key_version: "encryptionkey@0"
I20260926 02:58:23.063841 9610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:23.086114 9610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:23.090680 9610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:23.091017 9610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:23.110284 9610 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:33987
I20260926 02:58:23.110316 9676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:33987 every 8 connection(s)
I20260926 02:58:23.111739 9610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:23.115654 9677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:23.119374 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 9610
I20260926 02:58:23.119567 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:23.120134 20199 external_mini_cluster.cc:1428] Setting key 0e2b7a1fbac8035cee7b96c7c32bb291
I20260926 02:58:23.128878 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0: Bootstrap starting.
I20260926 02:58:23.132728 9677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:23.133939 9677 log.cc:824] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:23.134083 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:23.122020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46877 (local address 127.19.185.254:33987)
0926 02:58:23.122852 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:23.122865 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:23.122907 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:58:23.124622 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:23.124648 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:23.125071 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:23.125407 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:23.126112 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:23.127537 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:23.129415 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:23.130097 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:23.130224 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:23.131674 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:23.131682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:23.131696 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:23.132235 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:23.132313 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:23.132338 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:23.132489 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:23.132815 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 02:58:23.133028 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":415,"threads_started":1}
I20260926 02:58:23.137022 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0: No bootstrap required, opened a new log
I20260926 02:58:23.137946 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:23.146019 9677 raft_consensus.cc:348] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f469637a92f2448da318819e7ee58bb0" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 33987 } }
I20260926 02:58:23.146490 9677 raft_consensus.cc:374] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:23.146624 9677 raft_consensus.cc:729] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f469637a92f2448da318819e7ee58bb0, State: Initialized, Role: FOLLOWER
I20260926 02:58:23.147272 9677 consensus_queue.cc:260] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f469637a92f2448da318819e7ee58bb0" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 33987 } }
I20260926 02:58:23.147600 9677 raft_consensus.cc:388] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:23.147778 9677 raft_consensus.cc:482] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:23.147975 9677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:23.148870 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:23.142290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43533 (local address 127.19.185.254:33987)
0926 02:58:23.142454 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:23.142459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:23.142913 (+ 454us) server_negotiation.cc:407] Connection header received
0926 02:58:23.143125 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:23.143131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:23.143206 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:23.143336 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:23.143859 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:23.144663 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:23.146040 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:23.146644 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:23.146817 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:23.148014 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:23.148019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:23.148022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:23.148039 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:23.148121 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:23.148142 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:23.148231 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:23.148464 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 02:58:23.148506 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 02:58:23.150604 9677 raft_consensus.cc:504] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f469637a92f2448da318819e7ee58bb0" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 33987 } }
I20260926 02:58:23.151115 9677 leader_election.cc:302] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f469637a92f2448da318819e7ee58bb0; no voters:
I20260926 02:58:23.151749 9677 leader_election.cc:288] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:23.151888 9687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:23.152578 9687 raft_consensus.cc:686] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [term 1 LEADER]: Becoming Leader. State: Replica: f469637a92f2448da318819e7ee58bb0, State: Running, Role: LEADER
I20260926 02:58:23.153157 9687 consensus_queue.cc:237] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f469637a92f2448da318819e7ee58bb0" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 33987 } }
I20260926 02:58:23.154307 9677 sys_catalog.cc:564] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:23.156441 9689 sys_catalog.cc:455] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f469637a92f2448da318819e7ee58bb0. Latest consensus state: current_term: 1 leader_uuid: "f469637a92f2448da318819e7ee58bb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f469637a92f2448da318819e7ee58bb0" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 33987 } } }
I20260926 02:58:23.156471 9688 sys_catalog.cc:455] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f469637a92f2448da318819e7ee58bb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f469637a92f2448da318819e7ee58bb0" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 33987 } } }
I20260926 02:58:23.157030 9688 sys_catalog.cc:458] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:23.157034 9689 sys_catalog.cc:458] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:23.158480 9693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:23.163784 9693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:23.176349 9693 catalog_manager.cc:1347] Generated new cluster ID: 65868025dbd144a69c062995daa870db
I20260926 02:58:23.176491 9693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:23.224888 9693 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:23.226210 9693 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:23.238855 9693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f469637a92f2448da318819e7ee58bb0: Generated new TSK 0
I20260926 02:58:23.239818 9693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:23.254941 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9610
W20260926 02:58:23.256695 9672 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46877
W20260926 02:58:23.257462 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9610: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:58:23.257613 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 9610
W20260926 02:58:23.274923 9595 connection.cc:535] client connection to 127.19.185.254:33987 recv error: Network error: failed to read from TLS socket (remote: 127.19.185.254:33987): Connection reset by peer (error 104)
2026-09-26T02:58:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1963 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:23 dist-test-slave-ktng krb5kdc[9708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:23 dist-test-slave-ktng krb5kdc[9708](info): set up 2 sockets
Sep 26 02:58:23 dist-test-slave-ktng krb5kdc[9708](info): commencing operation
krb5kdc: starting...
W20260926 02:58:24.380872 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:24 dist-test-slave-ktng krb5kdc[9708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:58:24.648692 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:39891
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:39891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:24.989207 9721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:24.989504 9721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:24.989661 9721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:24.997146 9721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:24.997267 9721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:24.997347 9721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:24.997440 9721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:24.997519 9721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:25.007742 9721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:39891
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:39891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:25.009104 9721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:25.010417 9721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:25.022394 9726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:25.022470 9727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:25.022598 9729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:25.023653 9721 server_base.cc:1034] running on GCE node
I20260926 02:58:25.024422 9721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:25.025904 9721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:25.027189 9721 hybrid_clock.cc:648] HybridClock initialized: now 1790391505027129 us; error 57 us; skew 500 ppm
I20260926 02:58:25.027592 9721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:25 dist-test-slave-ktng krb5kdc[9708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391505, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:25.030880 9721 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:58:25.032421 9721 webserver.cc:460] Webserver started at http://127.19.185.254:38333/ using document root <none> and password file <none>
I20260926 02:58:25.033114 9721 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:25.033218 9721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:25.033484 9721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:25.036616 9721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "e3f8856b54c243c88012ce70c87a9a19"
format_stamp: "Formatted at 2026-09-26 02:58:25 on dist-test-slave-ktng"
server_key: "757c4808280a85847a7e639c8de11366"
server_key_iv: "7cf942e0619bc2fb89457f581ce083ff"
server_key_version: "encryptionkey@0"
I20260926 02:58:25.037353 9721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "e3f8856b54c243c88012ce70c87a9a19"
format_stamp: "Formatted at 2026-09-26 02:58:25 on dist-test-slave-ktng"
server_key: "757c4808280a85847a7e639c8de11366"
server_key_iv: "7cf942e0619bc2fb89457f581ce083ff"
server_key_version: "encryptionkey@0"
I20260926 02:58:25.042775 9721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 02:58:25.046857 9736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:25.048444 9721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:58:25.048631 9721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "e3f8856b54c243c88012ce70c87a9a19"
format_stamp: "Formatted at 2026-09-26 02:58:25 on dist-test-slave-ktng"
server_key: "757c4808280a85847a7e639c8de11366"
server_key_iv: "7cf942e0619bc2fb89457f581ce083ff"
server_key_version: "encryptionkey@0"
I20260926 02:58:25.048878 9721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:25.054828 9721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:25.059010 9721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:25.059345 9721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:25.078035 9721 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:39891
I20260926 02:58:25.078158 9787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:39891 every 8 connection(s)
I20260926 02:58:25.079579 9721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:25.083787 9788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:25.084129 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 9721
I20260926 02:58:25.084306 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:25.084868 20199 external_mini_cluster.cc:1428] Setting key 5f5662220220afae505449b6a7cb394c
I20260926 02:58:25.092525 9791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:25.086665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33037 (local address 127.19.185.254:39891)
0926 02:58:25.087576 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:25.087588 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:25.087636 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:58:25.089123 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:25.089145 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:25.089507 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:25.089858 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:25.090914 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:25.090920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:25.090932 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:25.090947 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:25.091006 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:25.091026 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:25.091209 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:25.091363 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 02:58:25.091575 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":411,"threads_started":1}
I20260926 02:58:25.096077 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:25.097596 9788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19: Bootstrap starting.
I20260926 02:58:25.100679 9788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:25.101908 9788 log.cc:824] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:25.103735 9791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:25.100902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50947 (local address 127.19.185.254:39891)
0926 02:58:25.101101 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:25.101109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:25.101644 (+ 535us) server_negotiation.cc:407] Connection header received
0926 02:58:25.101793 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:25.101798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:25.101862 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:25.101981 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:25.103029 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:25.103033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:25.103036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:25.103053 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:25.103088 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:25.103102 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:25.103187 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:25.103412 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 02:58:25.103473 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:58:25.104998 9788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19: No bootstrap required, opened a new log
I20260926 02:58:25.112165 9788 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f8856b54c243c88012ce70c87a9a19" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39891 } }
I20260926 02:58:25.112542 9788 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:25.112653 9788 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f8856b54c243c88012ce70c87a9a19, State: Initialized, Role: FOLLOWER
I20260926 02:58:25.113204 9788 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f8856b54c243c88012ce70c87a9a19" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39891 } }
I20260926 02:58:25.113448 9788 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:25.113587 9788 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:25.113754 9788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:25.115901 9788 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f8856b54c243c88012ce70c87a9a19" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39891 } }
I20260926 02:58:25.116376 9788 leader_election.cc:302] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3f8856b54c243c88012ce70c87a9a19; no voters:
I20260926 02:58:25.116950 9788 leader_election.cc:288] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:25.117098 9798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:25.118502 9798 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [term 1 LEADER]: Becoming Leader. State: Replica: e3f8856b54c243c88012ce70c87a9a19, State: Running, Role: LEADER
I20260926 02:58:25.119005 9798 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f8856b54c243c88012ce70c87a9a19" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39891 } }
I20260926 02:58:25.119261 9788 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:25.122188 9800 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3f8856b54c243c88012ce70c87a9a19. Latest consensus state: current_term: 1 leader_uuid: "e3f8856b54c243c88012ce70c87a9a19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f8856b54c243c88012ce70c87a9a19" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39891 } } }
I20260926 02:58:25.122690 9800 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:25.123255 9799 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3f8856b54c243c88012ce70c87a9a19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f8856b54c243c88012ce70c87a9a19" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 39891 } } }
I20260926 02:58:25.123627 9799 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:25.128137 9807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:25.132781 9807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:25.141126 9807 catalog_manager.cc:1347] Generated new cluster ID: 220e5a34432947179b7fadeff6fd275b
I20260926 02:58:25.141286 9807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:25.152685 9807 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:25.153206 9807 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:25.170785 9807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3f8856b54c243c88012ce70c87a9a19: Generated new TSK 0
I20260926 02:58:25.171751 9807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:25.186385 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9721
W20260926 02:58:25.187951 9783 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33037
W20260926 02:58:25.188622 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9721: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:58:25.188796 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 9721
2026-09-26T02:58:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:25 dist-test-slave-ktng krb5kdc[9819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:25 dist-test-slave-ktng krb5kdc[9819](info): set up 2 sockets
Sep 26 02:58:25 dist-test-slave-ktng krb5kdc[9819](info): commencing operation
krb5kdc: starting...
W20260926 02:58:26.305281 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:26 dist-test-slave-ktng krb5kdc[9819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:58:26.575742 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:36753
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:36753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:26.908080 9832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:26.908473 9832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:26.908640 9832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:26.915917 9832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:26.916075 9832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:26.916180 9832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:26.916285 9832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:26.916368 9832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:26.926425 9832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:36753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:36753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:26.927781 9832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:26.929095 9832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:26.942143 9837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:26.942202 9838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:26.942265 9840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:26.942499 9832 server_base.cc:1034] running on GCE node
I20260926 02:58:26.943624 9832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:26.945016 9832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:26.946256 9832 hybrid_clock.cc:648] HybridClock initialized: now 1790391506946199 us; error 45 us; skew 500 ppm
I20260926 02:58:26.946622 9832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:26 dist-test-slave-ktng krb5kdc[9819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391506, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:26.955631 9832 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:58:26.957175 9832 webserver.cc:460] Webserver started at http://127.19.185.254:44001/ using document root <none> and password file <none>
I20260926 02:58:26.957922 9832 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:26.958081 9832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:26.958387 9832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:26.961480 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "d5dcda11b58346a5a03abb7c8a85939f"
format_stamp: "Formatted at 2026-09-26 02:58:26 on dist-test-slave-ktng"
server_key: "1e493e021dc1a99fc442618533917888"
server_key_iv: "31ec74961aff2aedc5b38974da3ee268"
server_key_version: "encryptionkey@0"
I20260926 02:58:26.962271 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "d5dcda11b58346a5a03abb7c8a85939f"
format_stamp: "Formatted at 2026-09-26 02:58:26 on dist-test-slave-ktng"
server_key: "1e493e021dc1a99fc442618533917888"
server_key_iv: "31ec74961aff2aedc5b38974da3ee268"
server_key_version: "encryptionkey@0"
I20260926 02:58:26.967741 9832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 02:58:26.971287 9847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:26.972867 9832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:58:26.973058 9832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "d5dcda11b58346a5a03abb7c8a85939f"
format_stamp: "Formatted at 2026-09-26 02:58:26 on dist-test-slave-ktng"
server_key: "1e493e021dc1a99fc442618533917888"
server_key_iv: "31ec74961aff2aedc5b38974da3ee268"
server_key_version: "encryptionkey@0"
I20260926 02:58:26.973258 9832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:26.991546 9832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:26.996138 9832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:26.996480 9832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:27.015632 9832 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:36753
I20260926 02:58:27.015669 9898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:36753 every 8 connection(s)
I20260926 02:58:27.017046 9832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:27.020825 9899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:27.021100 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 9832
I20260926 02:58:27.021270 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:27.021687 20199 external_mini_cluster.cc:1428] Setting key 3463142837eb83b5ee684baf19bb52a2
I20260926 02:58:27.034366 9899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f: Bootstrap starting.
Sep 26 02:58:27 dist-test-slave-ktng krb5kdc[9819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:58:27.038020 9899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:27.039104 9899 log.cc:824] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:27.042376 9899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f: No bootstrap required, opened a new log
I20260926 02:58:27.046213 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:27.023428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51349 (local address 127.19.185.254:36753)
0926 02:58:27.024220 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:27.024230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:27.024277 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:58:27.026542 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:27.026567 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:27.026953 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:27.027267 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:27.029005 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:27.030344 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:27.032240 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:27.032934 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:27.033068 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:27.037851 (+ 4783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:27.037859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:27.037873 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:27.037909 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:27.041629 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:27.042311 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:27.042330 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:27.042343 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:27.042548 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:27.042916 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:27.042921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:27.042924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:27.043649 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:27.043941 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:27.044380 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 02:58:27.044830 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":357,"threads_started":1}
I20260926 02:58:27.050050 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:27.051864 9899 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5dcda11b58346a5a03abb7c8a85939f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36753 } }
I20260926 02:58:27.052204 9899 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:27.052294 9899 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5dcda11b58346a5a03abb7c8a85939f, State: Initialized, Role: FOLLOWER
I20260926 02:58:27.052902 9899 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5dcda11b58346a5a03abb7c8a85939f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36753 } }
I20260926 02:58:27.053138 9899 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:27.053283 9899 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:27.053467 9899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:27.056272 9899 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5dcda11b58346a5a03abb7c8a85939f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36753 } }
I20260926 02:58:27.056770 9899 leader_election.cc:302] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5dcda11b58346a5a03abb7c8a85939f; no voters:
I20260926 02:58:27.057440 9899 leader_election.cc:288] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:27.057557 9909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:27.059670 9899 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:27.063956 9909 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [term 1 LEADER]: Becoming Leader. State: Replica: d5dcda11b58346a5a03abb7c8a85939f, State: Running, Role: LEADER
I20260926 02:58:27.064688 9909 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5dcda11b58346a5a03abb7c8a85939f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36753 } }
I20260926 02:58:27.067556 9910 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5dcda11b58346a5a03abb7c8a85939f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5dcda11b58346a5a03abb7c8a85939f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36753 } } }
I20260926 02:58:27.068048 9910 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:27.071544 9911 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5dcda11b58346a5a03abb7c8a85939f. Latest consensus state: current_term: 1 leader_uuid: "d5dcda11b58346a5a03abb7c8a85939f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5dcda11b58346a5a03abb7c8a85939f" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 36753 } } }
I20260926 02:58:27.072010 9911 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:27.072877 9915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:27.074954 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:27.054398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35589 (local address 127.19.185.254:36753)
0926 02:58:27.054577 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:27.054583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:27.055104 (+ 521us) server_negotiation.cc:407] Connection header received
0926 02:58:27.055241 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:27.055246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:27.055367 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:27.055475 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:27.056868 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:27.058104 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:27.059991 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:27.060694 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:27.060841 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:27.064739 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:27.064750 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:27.064758 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:27.064814 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:27.070216 (+ 5402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:27.071415 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:27.071422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:27.071428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:27.071609 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:27.072166 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:27.072174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:27.072180 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:27.072713 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:27.073955 (+ 1242us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:27.074138 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 02:58:27.074436 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:58:27.078886 9915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:27.088229 9915 catalog_manager.cc:1347] Generated new cluster ID: b18816fa1b65473d89baa609b7ee4c68
I20260926 02:58:27.088378 9915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:27.106495 9915 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:27.108063 9915 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:27.117043 9915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5dcda11b58346a5a03abb7c8a85939f: Generated new TSK 0
I20260926 02:58:27.118124 9915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:27.130555 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9832
I20260926 02:58:27.131918 9894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:58:27.228415 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 9832
2026-09-26T02:58:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2038 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:27 dist-test-slave-ktng krb5kdc[9931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:27 dist-test-slave-ktng krb5kdc[9931](info): set up 2 sockets
Sep 26 02:58:27 dist-test-slave-ktng krb5kdc[9931](info): commencing operation
krb5kdc: starting...
W20260926 02:58:28.340706 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:28 dist-test-slave-ktng krb5kdc[9931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:58:28.610839 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:44179
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:28.946313 9944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:28.946590 9944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:28.946772 9944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:28.953886 9944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:28.954015 9944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:28.954111 9944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:28.954191 9944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:28.954267 9944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:28.964145 9944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:44179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:44179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:28.965545 9944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:28.966871 9944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:28.978473 9950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:28.978469 9952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:28.978657 9949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:28.979632 9944 server_base.cc:1034] running on GCE node
I20260926 02:58:28.980425 9944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:28.981784 9944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:28.983018 9944 hybrid_clock.cc:648] HybridClock initialized: now 1790391508982977 us; error 33 us; skew 500 ppm
I20260926 02:58:28.983422 9944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:28 dist-test-slave-ktng krb5kdc[9931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391508, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:28.986447 9944 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:58:28.988005 9944 webserver.cc:460] Webserver started at http://127.19.185.254:38823/ using document root <none> and password file <none>
I20260926 02:58:28.988714 9944 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:28.988965 9944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:28.989318 9944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:28.992384 9944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "b6d2dc3c9003468bb159c26429058f0c"
format_stamp: "Formatted at 2026-09-26 02:58:28 on dist-test-slave-ktng"
server_key: "bb93c2f71b0fc31b0eaad6aef75aa7d1"
server_key_iv: "af2649f9b34a7ba651199e0874e0c647"
server_key_version: "encryptionkey@0"
I20260926 02:58:28.993131 9944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "b6d2dc3c9003468bb159c26429058f0c"
format_stamp: "Formatted at 2026-09-26 02:58:28 on dist-test-slave-ktng"
server_key: "bb93c2f71b0fc31b0eaad6aef75aa7d1"
server_key_iv: "af2649f9b34a7ba651199e0874e0c647"
server_key_version: "encryptionkey@0"
I20260926 02:58:28.998458 9944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:58:29.002192 9959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:29.003697 9944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:58:29.003898 9944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "b6d2dc3c9003468bb159c26429058f0c"
format_stamp: "Formatted at 2026-09-26 02:58:28 on dist-test-slave-ktng"
server_key: "bb93c2f71b0fc31b0eaad6aef75aa7d1"
server_key_iv: "af2649f9b34a7ba651199e0874e0c647"
server_key_version: "encryptionkey@0"
I20260926 02:58:29.004110 9944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:29.017995 9944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:29.022136 9944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:29.022439 9944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:29.041441 9944 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:44179
I20260926 02:58:29.041478 10010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:44179 every 8 connection(s)
I20260926 02:58:29.042869 9944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:29.046623 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 9944
I20260926 02:58:29.046819 10011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:29.046835 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:29.047349 20199 external_mini_cluster.cc:1428] Setting key 91b9e8dd3125e9312480fc84dd708dfb
I20260926 02:58:29.060885 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:29.049128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34117 (local address 127.19.185.254:44179)
0926 02:58:29.050019 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:29.050033 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:29.050075 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:58:29.051660 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:29.051681 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:29.052087 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:29.052418 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:29.053159 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:29.054476 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:29.056301 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:29.056970 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:29.057114 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:29.058409 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:29.058417 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:29.058430 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:29.059010 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:29.059091 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:29.059115 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:29.059201 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:29.059479 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 02:58:29.059741 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":396,"threads_started":1}
I20260926 02:58:29.061185 10011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c: Bootstrap starting.
I20260926 02:58:29.064638 10011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:29.064888 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:29.065758 10011 log.cc:824] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:29.068630 10011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c: No bootstrap required, opened a new log
I20260926 02:58:29.076272 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:29.069461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41893 (local address 127.19.185.254:44179)
0926 02:58:29.069628 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:29.069634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:29.070056 (+ 422us) server_negotiation.cc:407] Connection header received
0926 02:58:29.070210 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:29.070216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:29.070304 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:29.070423 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:29.071073 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:29.072253 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:29.073611 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:29.074321 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:29.074455 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:29.075521 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:29.075531 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:29.075536 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:29.075568 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:29.075618 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:29.075644 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:29.075751 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:29.075852 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 02:58:29.075941 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:58:29.078255 10011 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d2dc3c9003468bb159c26429058f0c" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44179 } }
I20260926 02:58:29.078666 10011 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:29.078785 10011 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d2dc3c9003468bb159c26429058f0c, State: Initialized, Role: FOLLOWER
I20260926 02:58:29.079439 10011 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d2dc3c9003468bb159c26429058f0c" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44179 } }
I20260926 02:58:29.079702 10011 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:29.079870 10011 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:29.080053 10011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:29.082355 10011 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d2dc3c9003468bb159c26429058f0c" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44179 } }
I20260926 02:58:29.082808 10011 leader_election.cc:302] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6d2dc3c9003468bb159c26429058f0c; no voters:
I20260926 02:58:29.083376 10011 leader_election.cc:288] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:29.083516 10021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:29.084244 10021 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [term 1 LEADER]: Becoming Leader. State: Replica: b6d2dc3c9003468bb159c26429058f0c, State: Running, Role: LEADER
I20260926 02:58:29.084811 10021 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d2dc3c9003468bb159c26429058f0c" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44179 } }
I20260926 02:58:29.085525 10011 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:29.087563 10023 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6d2dc3c9003468bb159c26429058f0c. Latest consensus state: current_term: 1 leader_uuid: "b6d2dc3c9003468bb159c26429058f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d2dc3c9003468bb159c26429058f0c" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44179 } } }
I20260926 02:58:29.087910 10023 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:29.088809 10026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:29.088660 10022 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6d2dc3c9003468bb159c26429058f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d2dc3c9003468bb159c26429058f0c" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 44179 } } }
I20260926 02:58:29.088999 10022 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:29.095266 10026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:29.104111 10026 catalog_manager.cc:1347] Generated new cluster ID: 928761fb5c734530bd17139c864d4460
I20260926 02:58:29.104241 10026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:29.118264 10026 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:29.119827 10026 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:29.131840 10026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6d2dc3c9003468bb159c26429058f0c: Generated new TSK 0
I20260926 02:58:29.132735 10026 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:29.144824 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9944
W20260926 02:58:29.145893 10006 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34117
W20260926 02:58:29.146535 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 9944: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:58:29.146689 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 9944
2026-09-26T02:58:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1919 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:29 dist-test-slave-ktng krb5kdc[10042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:29 dist-test-slave-ktng krb5kdc[10042](info): set up 2 sockets
Sep 26 02:58:29 dist-test-slave-ktng krb5kdc[10042](info): commencing operation
krb5kdc: starting...
W20260926 02:58:30.281302 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:30 dist-test-slave-ktng krb5kdc[10042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:58:30.551340 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:43261
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:44271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:43261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:30.885393 10055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:30.885686 10055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:30.885905 10055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:30.893198 10055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:30.893311 10055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:30.893405 10055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:30.893510 10055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:30.893594 10055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:30.903540 10055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:44271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:43261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:43261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:30.904997 10055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:30.906369 10055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:30.918821 10063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:30.918900 10061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:30.918843 10060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:30.925091 10055 server_base.cc:1034] running on GCE node
I20260926 02:58:30.925951 10055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:30.927541 10055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:30.928784 10055 hybrid_clock.cc:648] HybridClock initialized: now 1790391510928750 us; error 33 us; skew 500 ppm
I20260926 02:58:30.929159 10055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:30 dist-test-slave-ktng krb5kdc[10042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391510, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:30.932435 10055 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:58:30.934062 10055 webserver.cc:460] Webserver started at http://127.19.185.254:46505/ using document root <none> and password file <none>
I20260926 02:58:30.934804 10055 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:30.934955 10055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:30.935263 10055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:30.938308 10055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "361bb480869d47aa8f2d6d2fc72ac57a"
format_stamp: "Formatted at 2026-09-26 02:58:30 on dist-test-slave-ktng"
server_key: "d7ddf029dda315fcebef91d404e54b2a"
server_key_iv: "ef29411f4151e8bd7cb1db491f6ca0a9"
server_key_version: "encryptionkey@0"
I20260926 02:58:30.939088 10055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "361bb480869d47aa8f2d6d2fc72ac57a"
format_stamp: "Formatted at 2026-09-26 02:58:30 on dist-test-slave-ktng"
server_key: "d7ddf029dda315fcebef91d404e54b2a"
server_key_iv: "ef29411f4151e8bd7cb1db491f6ca0a9"
server_key_version: "encryptionkey@0"
I20260926 02:58:30.944545 10055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:58:30.948117 10070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:30.949489 10055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:58:30.949680 10055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "361bb480869d47aa8f2d6d2fc72ac57a"
format_stamp: "Formatted at 2026-09-26 02:58:30 on dist-test-slave-ktng"
server_key: "d7ddf029dda315fcebef91d404e54b2a"
server_key_iv: "ef29411f4151e8bd7cb1db491f6ca0a9"
server_key_version: "encryptionkey@0"
I20260926 02:58:30.949883 10055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:30.955714 10055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:30.960139 10055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:30.960448 10055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:30.980263 10055 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:43261
I20260926 02:58:30.980324 10121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:43261 every 8 connection(s)
I20260926 02:58:30.981742 10055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:30.985766 10122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:30.987255 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 10055
I20260926 02:58:30.987567 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:30.988022 20199 external_mini_cluster.cc:1428] Setting key fdf7da03f7893fd6c1c5bbfe2ecf6100
I20260926 02:58:30.996210 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:30.989891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60667 (local address 127.19.185.254:43261)
0926 02:58:30.990782 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:30.990795 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:30.990837 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:58:30.992562 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:30.992586 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:30.992988 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:30.993347 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:30.994499 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:30.994505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:30.994518 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:30.994533 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:30.994593 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:30.994617 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:30.994782 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:30.994973 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 02:58:30.995197 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":440,"threads_started":1}
I20260926 02:58:31.000164 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:31.000379 10122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a: Bootstrap starting.
I20260926 02:58:31.003557 10122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:31.004748 10122 log.cc:824] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:31.007181 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:31.004664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50487 (local address 127.19.185.254:43261)
0926 02:58:31.004895 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:31.004900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:31.005206 (+ 306us) server_negotiation.cc:407] Connection header received
0926 02:58:31.005428 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:31.005436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:31.005507 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:31.005641 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:31.006551 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:31.006556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:31.006560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:58:31.006577 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:31.006619 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:58:31.006640 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:31.006756 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:31.006836 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 02:58:31.006891 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 02:58:31.008231 10122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a: No bootstrap required, opened a new log
I20260926 02:58:31.015580 10122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361bb480869d47aa8f2d6d2fc72ac57a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43261 } }
I20260926 02:58:31.015951 10122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:58:31.016024 10122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 361bb480869d47aa8f2d6d2fc72ac57a, State: Initialized, Role: FOLLOWER
I20260926 02:58:31.016531 10122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361bb480869d47aa8f2d6d2fc72ac57a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43261 } }
I20260926 02:58:31.016726 10122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:58:31.016824 10122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:58:31.016948 10122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:31.019069 10122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361bb480869d47aa8f2d6d2fc72ac57a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43261 } }
I20260926 02:58:31.019626 10122 leader_election.cc:302] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 361bb480869d47aa8f2d6d2fc72ac57a; no voters:
I20260926 02:58:31.020428 10122 leader_election.cc:288] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:58:31.020572 10132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:31.021242 10132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [term 1 LEADER]: Becoming Leader. State: Replica: 361bb480869d47aa8f2d6d2fc72ac57a, State: Running, Role: LEADER
I20260926 02:58:31.021783 10132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361bb480869d47aa8f2d6d2fc72ac57a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43261 } }
I20260926 02:58:31.022472 10122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:31.024999 10133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "361bb480869d47aa8f2d6d2fc72ac57a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361bb480869d47aa8f2d6d2fc72ac57a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43261 } } }
I20260926 02:58:31.025527 10133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:31.025096 10134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 361bb480869d47aa8f2d6d2fc72ac57a. Latest consensus state: current_term: 1 leader_uuid: "361bb480869d47aa8f2d6d2fc72ac57a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361bb480869d47aa8f2d6d2fc72ac57a" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 43261 } } }
I20260926 02:58:31.025981 10134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:31.027091 10141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:31.032618 10141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:31.041917 10141 catalog_manager.cc:1347] Generated new cluster ID: 4795b9760e1f4719a82950802a840671
I20260926 02:58:31.042047 10141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:31.059533 10141 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:31.060053 10141 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:31.078464 10141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 361bb480869d47aa8f2d6d2fc72ac57a: Generated new TSK 0
I20260926 02:58:31.079608 10141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:31.096688 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 10055
W20260926 02:58:31.097939 10117 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60667
W20260926 02:58:31.098678 20199 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 10055: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:58:31.098847 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 10055
2026-09-26T02:58:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1947 ms)
[----------] 18 tests from AuthTokenIssuingTest (35863 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:31 dist-test-slave-ktng krb5kdc[10153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:31 dist-test-slave-ktng krb5kdc[10153](info): set up 2 sockets
Sep 26 02:58:31 dist-test-slave-ktng krb5kdc[10153](info): commencing operation
krb5kdc: starting...
W20260926 02:58:32.212747 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:32 dist-test-slave-ktng krb5kdc[10153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:58:32.485138 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:41873
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:41873,127.19.185.253:39519,127.19.185.252:36349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:32.821179 10166 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:58:32.821599 10166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:32.821727 10166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:32.821873 10166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:32.829190 10166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:32.829339 10166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:32.829444 10166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:32.829540 10166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:32.829631 10166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:32.839740 10166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:41873,127.19.185.253:39519,127.19.185.252:36349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:32.841077 10166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:32.842409 10166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:32.854270 10171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:32.854300 10174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:32.854236 10172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:32.854998 10166 server_base.cc:1034] running on GCE node
I20260926 02:58:32.855873 10166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:32.857335 10166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:32.858559 10166 hybrid_clock.cc:648] HybridClock initialized: now 1790391512858502 us; error 29 us; skew 500 ppm
I20260926 02:58:32.858898 10166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:32 dist-test-slave-ktng krb5kdc[10153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391512, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:32.867785 10166 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:58:32.869984 10166 webserver.cc:460] Webserver started at http://127.19.185.254:36133/ using document root <none> and password file <none>
I20260926 02:58:32.871104 10166 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:32.871246 10166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:32.871707 10166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:32.874889 10166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "67e44e9e35984f33a1c0fb44c27e8391"
format_stamp: "Formatted at 2026-09-26 02:58:32 on dist-test-slave-ktng"
server_key: "aa9e8f44812ffa3d3d8c3e0c6212b58b"
server_key_iv: "2882ef91e37b14fa7d30b725ca61d1b7"
server_key_version: "encryptionkey@0"
I20260926 02:58:32.875741 10166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "67e44e9e35984f33a1c0fb44c27e8391"
format_stamp: "Formatted at 2026-09-26 02:58:32 on dist-test-slave-ktng"
server_key: "aa9e8f44812ffa3d3d8c3e0c6212b58b"
server_key_iv: "2882ef91e37b14fa7d30b725ca61d1b7"
server_key_version: "encryptionkey@0"
I20260926 02:58:32.881096 10166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:58:32.884639 10181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:32.885982 10166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:58:32.886158 10166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "67e44e9e35984f33a1c0fb44c27e8391"
format_stamp: "Formatted at 2026-09-26 02:58:32 on dist-test-slave-ktng"
server_key: "aa9e8f44812ffa3d3d8c3e0c6212b58b"
server_key_iv: "2882ef91e37b14fa7d30b725ca61d1b7"
server_key_version: "encryptionkey@0"
I20260926 02:58:32.886385 10166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:32.908636 10166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:32.912909 10166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:32.913187 10166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:32.932248 10166 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:41873
I20260926 02:58:32.932256 10232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:41873 every 8 connection(s)
I20260926 02:58:32.933645 10166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:32.937280 10233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:32.942255 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 10166
I20260926 02:58:32.942490 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:32.943153 20199 external_mini_cluster.cc:1428] Setting key 80b4a56eab05d01717a6142648389fa1
I20260926 02:58:32.943935 10233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:58:32 dist-test-slave-ktng krb5kdc[10153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391512, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:58:32.968492 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:32.947092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:41873 (local address 127.0.0.1:58792)
0926 02:58:32.948093 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:32.948169 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:32.950083 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:32.951043 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:32.951048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:32.951689 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:32.952032 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:32.952049 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:32.953844 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:32.953851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:32.954983 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:32.955000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:32.956461 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:32.956466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:32.956598 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:32.957367 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:32.957416 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:32.959651 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:32.964087 (+ 4436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:32.964097 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:32.964106 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:32.964539 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:32.965032 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:32.965035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:32.965038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:32.965180 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:32.966274 (+ 1094us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:32.966280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:32.966501 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 02:58:32.966816 (+ 315us) client_negotiation.cc:240] Negotiation successful
0926 02:58:32.967370 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":407,"threads_started":1}
I20260926 02:58:32.968545 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:32.947084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.19.185.254:41873)
0926 02:58:32.948797 (+ 1713us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:32.948810 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:32.948885 (+ 75us) server_negotiation.cc:407] Connection header received
0926 02:58:32.950481 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:32.950491 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:32.950794 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:32.951027 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:32.952257 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:32.953675 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:32.955137 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:32.956197 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:32.956405 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:32.959894 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:32.959907 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:32.959926 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:32.959973 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:32.963950 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:32.964695 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:32.964707 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:32.964714 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:32.964919 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:32.965307 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:32.965315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:32.965322 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:32.966106 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:32.966330 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:32.967017 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 02:58:32.967552 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":183,"server-negotiator.queue_time_us":1480,"spinlock_wait_cycles":17792,"thread_start_us":1286,"threads_started":1}
WARNING: no policy specified for kudu/127.19.185.253@KRBTEST.COM; defaulting to no policy
I20260926 02:58:32.971777 10233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:58:32.972886 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:32.972467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:39519 (local address 127.0.0.1:57076)
0926 02:58:32.972639 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:32.972757 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 02:58:32.973263 10183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.253@KRBTEST.COM" created.
I20260926 02:58:32.975231 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:32.974820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:39519 (local address 127.0.0.1:57084)
0926 02:58:32.975020 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:32.975108 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 02:58:32.975538 10233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:39519: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.253.keytab.
Entry for principal kudu/127.19.185.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.253.keytab.
I20260926 02:58:33.009069 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.253:39519
--webserver_interface=127.19.185.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.253.keytab
--principal=kudu/127.19.185.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:41873,127.19.185.253:39519,127.19.185.252:36349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:58:33.024878 10233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } attempt: 1
I20260926 02:58:33.026204 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.025689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:39519 (local address 127.0.0.1:57092)
0926 02:58:33.025947 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.026060 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 02:58:33.027213 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.026860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:39519 (local address 127.0.0.1:57098)
0926 02:58:33.027006 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.027081 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 02:58:33.027410 10233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:39519: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
I20260926 02:58:33.095649 10233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } attempt: 2
I20260926 02:58:33.096925 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.096427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:39519 (local address 127.0.0.1:57114)
0926 02:58:33.096697 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.096781 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 02:58:33.097914 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.097547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:39519 (local address 127.0.0.1:57120)
0926 02:58:33.097704 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.097785 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 02:58:33.098083 10233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:39519: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
I20260926 02:58:33.189287 10233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } attempt: 3
I20260926 02:58:33.190454 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.189947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:39519 (local address 127.0.0.1:57132)
0926 02:58:33.190140 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.190237 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 02:58:33.191515 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.191103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:39519 (local address 127.0.0.1:57146)
0926 02:58:33.191247 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.191371 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 02:58:33.191696 10233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:39519: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
I20260926 02:58:33.334961 10233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } attempt: 4
I20260926 02:58:33.336236 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.335700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:39519 (local address 127.0.0.1:57158)
0926 02:58:33.336017 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.336105 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260926 02:58:33.337219 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.336868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:39519 (local address 127.0.0.1:57168)
0926 02:58:33.337017 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.337100 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 02:58:33.337415 10233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:39519: Network error: Client connection negotiation failed: client connection to 127.19.185.253:39519: connect: Connection refused (error 111)
W20260926 02:58:33.348240 10240 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:58:33.348651 10240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:33.348775 10240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:33.348919 10240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:33.356381 10240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:33.356506 10240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:33.356631 10240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:33.356717 10240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:33.356827 10240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:33.366858 10240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:41873,127.19.185.253:39519,127.19.185.252:36349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.253.keytab
--principal=kudu/127.19.185.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.253:39519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.185.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:33.368323 10240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:33.369647 10240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:33.381868 10246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:33.381987 10245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:33.381987 10248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:33.382912 10240 server_base.cc:1034] running on GCE node
I20260926 02:58:33.383810 10240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:33.385195 10240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:33.386440 10240 hybrid_clock.cc:648] HybridClock initialized: now 1790391513386380 us; error 41 us; skew 500 ppm
I20260926 02:58:33.386776 10240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:33 dist-test-slave-ktng krb5kdc[10153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:33.390146 10240 init.cc:377] Logged in from keytab as kudu/127.19.185.253@KRBTEST.COM (short username kudu)
I20260926 02:58:33.391809 10240 webserver.cc:460] Webserver started at http://127.19.185.253:44965/ using document root <none> and password file <none>
I20260926 02:58:33.392524 10240 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:33.392661 10240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:33.392997 10240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:33.396059 10240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/data/instance:
uuid: "0c97679031a84d669bcccec539d9792f"
format_stamp: "Formatted at 2026-09-26 02:58:33 on dist-test-slave-ktng"
server_key: "22343a863b6fabfc11d2367801197b8d"
server_key_iv: "dbb08ebd48cd8b06c042799d67fb16c7"
server_key_version: "encryptionkey@0"
I20260926 02:58:33.396860 10240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/wal/instance:
uuid: "0c97679031a84d669bcccec539d9792f"
format_stamp: "Formatted at 2026-09-26 02:58:33 on dist-test-slave-ktng"
server_key: "22343a863b6fabfc11d2367801197b8d"
server_key_iv: "dbb08ebd48cd8b06c042799d67fb16c7"
server_key_version: "encryptionkey@0"
I20260926 02:58:33.401990 10240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:58:33.405390 10255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:33.406723 10240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:58:33.406924 10240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/wal
uuid: "0c97679031a84d669bcccec539d9792f"
format_stamp: "Formatted at 2026-09-26 02:58:33 on dist-test-slave-ktng"
server_key: "22343a863b6fabfc11d2367801197b8d"
server_key_iv: "dbb08ebd48cd8b06c042799d67fb16c7"
server_key_version: "encryptionkey@0"
I20260926 02:58:33.407142 10240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:33.431555 10240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:33.435858 10240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:33.436156 10240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:33.455102 10240 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.253:39519
I20260926 02:58:33.455111 10306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.253:39519 every 8 connection(s)
I20260926 02:58:33.456534 10240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/data/info.pb
I20260926 02:58:33.459345 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 10240
I20260926 02:58:33.459551 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-1/wal/instance
I20260926 02:58:33.460001 20199 external_mini_cluster.cc:1428] Setting key 081e10ac114581d63bf81c522b3351a7
I20260926 02:58:33.460465 10307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:33.466962 10307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:58:33 dist-test-slave-ktng krb5kdc[10153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.253@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.185.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.252@KRBTEST.COM" created.
I20260926 02:58:33.489182 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.469722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.19.185.254:41873)
0926 02:58:33.469940 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:33.469945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:33.471730 (+ 1785us) server_negotiation.cc:407] Connection header received
0926 02:58:33.473699 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:33.473705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:33.473801 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:33.473923 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:33.475691 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.476413 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.478007 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.478436 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.478588 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:33.482531 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:33.482536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:33.482539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:33.482574 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:33.485551 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:33.486377 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:33.486382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:33.486385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:33.486509 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:33.486996 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:33.487003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:33.487008 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:33.487404 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:33.487541 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:33.488505 (+ 964us) server_negotiation.cc:299] Negotiation successful
0926 02:58:33.488765 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 02:58:33.489904 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.470475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:41873 (local address 127.0.0.1:58798)
0926 02:58:33.471551 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.471589 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:33.473410 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:33.474245 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:33.474260 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:33.474862 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:33.475494 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:33.475521 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.476583 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.476588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:33.477814 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:33.477826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.478682 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.478690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:33.478934 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:33.479711 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:33.479735 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:33.482286 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:33.485703 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:33.485710 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:33.485722 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:33.486243 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:33.486607 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:33.486610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:33.486612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:33.486783 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:33.487548 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:33.487557 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:33.488081 (+ 524us) client_negotiation.cc:769] Sending connection context
0926 02:58:33.488330 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 02:58:33.488729 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":526,"threads_started":1}
I20260926 02:58:33.491864 10307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:58:33 dist-test-slave-ktng krb5kdc[10153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.253@KRBTEST.COM for kudu/127.19.185.253@KRBTEST.COM
I20260926 02:58:33.508107 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.492601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:39519 (local address 127.0.0.1:57184)
0926 02:58:33.492818 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.492835 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:33.492934 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:33.493963 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:33.493967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:33.494415 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:33.494548 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:33.494556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.496003 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.496010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:33.496862 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:33.496874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.498062 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.498066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:33.498211 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:33.498847 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:33.498864 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:33.500930 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:33.505561 (+ 4631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:33.505565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:33.505567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:33.505976 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:33.506412 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:33.506415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:33.506418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:33.506535 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:33.507404 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:33.507411 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:33.507522 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 02:58:33.507635 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 02:58:33.507833 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 02:58:33.508802 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.492915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.19.185.253:39519)
0926 02:58:33.493546 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:33.493552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:33.493577 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:58:33.493620 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:33.493626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:33.493812 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:33.493981 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:33.494728 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.495872 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.497002 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.497900 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.498042 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:33.501164 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:33.501181 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:33.501204 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:33.501247 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:33.505397 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:33.506116 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:33.506127 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:33.506134 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:33.506280 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:33.506651 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:33.506659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:33.506665 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:33.507216 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:33.507487 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:33.507819 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 02:58:33.508117 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":330,"threads_started":1}
I20260926 02:58:33.511571 10307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:58:33.512710 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.512248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:41842)
0926 02:58:33.512417 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.512572 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 02:58:33.513024 10259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
I20260926 02:58:33.514559 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.514214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:41858)
0926 02:58:33.514392 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.514465 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 02:58:33.514784 10307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:36349: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.252.keytab.
Entry for principal kudu/127.19.185.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.252.keytab.
I20260926 02:58:33.526156 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.252:36349
--webserver_interface=127.19.185.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.252.keytab
--principal=kudu/127.19.185.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:41873,127.19.185.253:39519,127.19.185.252:36349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:58:33.564072 10307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } attempt: 1
I20260926 02:58:33.565306 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.564738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:41874)
0926 02:58:33.564990 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.565122 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 02:58:33.566385 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.565967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:41884)
0926 02:58:33.566152 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.566244 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 02:58:33.566591 10307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:36349: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
I20260926 02:58:33.634852 10307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } attempt: 2
I20260926 02:58:33.636271 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.635631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:41898)
0926 02:58:33.635907 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.636044 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 02:58:33.636641 10233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } attempt: 5
I20260926 02:58:33.637805 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.637119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:41910)
0926 02:58:33.637286 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.637640 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 02:58:33.638139 10307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:36349: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Sep 26 02:58:33 dist-test-slave-ktng krb5kdc[10153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391512, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for kudu/127.19.185.253@KRBTEST.COM
I20260926 02:58:33.651690 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.637362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:39519 (local address 127.0.0.1:57194)
0926 02:58:33.637570 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.637591 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:33.637779 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:33.638237 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:33.638244 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:33.638635 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:33.638823 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:33.638832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.640251 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.640256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:33.641199 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:33.641212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.642227 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.642232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:33.642416 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:33.643089 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:33.643109 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:33.645238 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:33.649048 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:33.649052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:33.649055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:33.649471 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:33.649898 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:33.649903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:33.649906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:33.650047 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:33.650615 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:33.650620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:33.650776 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 02:58:33.650998 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 02:58:33.651324 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:58:33.651685 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.637527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.19.185.253:39519)
0926 02:58:33.637691 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:33.637698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:33.637722 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:58:33.637938 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:33.637944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:33.638078 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:33.638191 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:33.638977 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.640096 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.641365 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.642063 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.642219 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:33.645441 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:33.645448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:33.645453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:33.645494 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:33.648895 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:33.649646 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:33.649655 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:33.649657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:33.649777 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:33.650156 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:33.650162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:33.650165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:33.650482 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:33.650653 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:33.651111 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 02:58:33.651349 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 02:58:33.652969 10233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:58:33.654086 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.653714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:41926)
0926 02:58:33.653879 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.653966 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 02:58:33.655049 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.654690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:41940)
0926 02:58:33.654839 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.654948 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 02:58:33.655272 10233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:36349: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
I20260926 02:58:33.706585 10233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } attempt: 1
I20260926 02:58:33.707926 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.707360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:41952)
0926 02:58:33.707601 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.707716 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 02:58:33.708945 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.708578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:41964)
0926 02:58:33.708733 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.708817 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 02:58:33.709170 10233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:36349: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
I20260926 02:58:33.729435 10307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } attempt: 3
I20260926 02:58:33.730604 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.730116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:41966)
0926 02:58:33.730333 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.730452 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 02:58:33.731814 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.731372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:41972)
0926 02:58:33.731626 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.731698 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260926 02:58:33.732080 10307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:36349: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
I20260926 02:58:33.777411 10233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } attempt: 2
I20260926 02:58:33.778746 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.778131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:41982)
0926 02:58:33.778373 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.778518 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 02:58:33.779964 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.779533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:41998)
0926 02:58:33.779709 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.779805 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 02:58:33.780238 10233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:36349: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
I20260926 02:58:33.875330 10307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } attempt: 4
I20260926 02:58:33.876521 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.876040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:42006)
0926 02:58:33.876266 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.876362 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 02:58:33.877512 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.877176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:42010)
0926 02:58:33.877326 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.877410 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 02:58:33.877745 10307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:36349: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
W20260926 02:58:33.877827 10314 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:58:33.878242 10314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:33.878372 10314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:33.878536 10314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 02:58:33.886515 10233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } attempt: 3
W20260926 02:58:33.886724 10314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:33.886829 10314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:33.886924 10314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:33.886999 10314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:33.887092 10314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:33.887763 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.887211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:42016)
0926 02:58:33.887480 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.887581 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 02:58:33.888851 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.888436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:42030)
0926 02:58:33.888630 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.888720 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260926 02:58:33.889055 10233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:36349: Network error: Client connection negotiation failed: client connection to 127.19.185.252:36349: connect: Connection refused (error 111)
I20260926 02:58:33.898168 10314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:41873,127.19.185.253:39519,127.19.185.252:36349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.252.keytab
--principal=kudu/127.19.185.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.252:36349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.185.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:33.899588 10314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:33.900947 10314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:33.912997 10320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:33.912943 10322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:33.913149 10319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:33.913717 10314 server_base.cc:1034] running on GCE node
I20260926 02:58:33.914428 10314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:33.915823 10314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:33.917228 10314 hybrid_clock.cc:648] HybridClock initialized: now 1790391513917118 us; error 82 us; skew 500 ppm
I20260926 02:58:33.917583 10314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:33 dist-test-slave-ktng krb5kdc[10153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:33.920836 10314 init.cc:377] Logged in from keytab as kudu/127.19.185.252@KRBTEST.COM (short username kudu)
I20260926 02:58:33.922397 10314 webserver.cc:460] Webserver started at http://127.19.185.252:46161/ using document root <none> and password file <none>
I20260926 02:58:33.923069 10314 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:33.923162 10314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:33.923503 10314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:33.926466 10314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/data/instance:
uuid: "a0775c0c2f504e8ea4a028a6b8ac948d"
format_stamp: "Formatted at 2026-09-26 02:58:33 on dist-test-slave-ktng"
server_key: "49432868d8b660082988a143396c12a1"
server_key_iv: "4f0217ccbe51c8aef0fcc016c07f8c2f"
server_key_version: "encryptionkey@0"
I20260926 02:58:33.927225 10314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/wal/instance:
uuid: "a0775c0c2f504e8ea4a028a6b8ac948d"
format_stamp: "Formatted at 2026-09-26 02:58:33 on dist-test-slave-ktng"
server_key: "49432868d8b660082988a143396c12a1"
server_key_iv: "4f0217ccbe51c8aef0fcc016c07f8c2f"
server_key_version: "encryptionkey@0"
I20260926 02:58:33.932775 10314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:58:33.937093 10329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:33.938772 10314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:58:33.938942 10314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/wal
uuid: "a0775c0c2f504e8ea4a028a6b8ac948d"
format_stamp: "Formatted at 2026-09-26 02:58:33 on dist-test-slave-ktng"
server_key: "49432868d8b660082988a143396c12a1"
server_key_iv: "4f0217ccbe51c8aef0fcc016c07f8c2f"
server_key_version: "encryptionkey@0"
I20260926 02:58:33.939163 10314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:33.950713 10314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:33.954720 10314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:33.954949 10314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:33.973187 10314 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.252:36349
I20260926 02:58:33.973207 10380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.252:36349 every 8 connection(s)
I20260926 02:58:33.974597 10314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/data/info.pb
I20260926 02:58:33.978631 10381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:33.982321 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 10314
I20260926 02:58:33.982512 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391151502216-20199-0/minicluster-data/master-2/wal/instance
I20260926 02:58:33.983073 20199 external_mini_cluster.cc:1428] Setting key 63690242f29c4a2203a28b691346388b
I20260926 02:58:33.986544 10381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:58:33 dist-test-slave-ktng krb5kdc[10153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.252@KRBTEST.COM
Sep 26 02:58:33 dist-test-slave-ktng krb5kdc[10153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.252@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:58:34.006943 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.990320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.19.185.254:41873)
0926 02:58:33.990993 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:33.990999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:33.991487 (+ 488us) server_negotiation.cc:407] Connection header received
0926 02:58:33.991674 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:33.991678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:33.991747 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:33.991888 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:33.993223 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.994386 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.996353 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.996965 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.997108 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.001108 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:34.001113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:34.001116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:34.001150 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:34.003900 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.004578 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.004582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.004584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.004707 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.005102 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.005108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.005111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.005396 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:34.005525 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.006195 (+ 670us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.006457 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":352,"threads_started":1}
I20260926 02:58:34.008040 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.985102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.19.185.252:36349)
0926 02:58:33.985953 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:33.985969 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:33.986012 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:58:33.987752 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:33.987775 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:33.988202 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:33.988621 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:33.990315 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.992608 (+ 2293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.994416 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.995629 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.995810 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.000501 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:34.000512 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:34.000528 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:34.000578 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:34.003745 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.004512 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.004522 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.004525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.004679 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.005132 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.005135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.005137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.005691 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:34.006016 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.006401 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.006621 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":548,"thread_start_us":317,"threads_started":1}
I20260926 02:58:34.008051 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:33.990411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:41873 (local address 127.0.0.1:58800)
0926 02:58:33.991340 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:33.991390 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:33.991554 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:33.991960 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:33.991967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:33.992822 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:33.993062 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:33.993070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.994559 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.994564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:33.996185 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:33.996195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:33.997179 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:33.997185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:33.997320 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:33.998021 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:33.998046 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:34.000768 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:34.004025 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:34.004033 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:34.004043 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:34.004428 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:34.004813 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:34.004817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:34.004821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:34.004980 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:34.005522 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:34.005529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:34.005747 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 02:58:34.006029 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 02:58:34.006534 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":476,"threads_started":1}
I20260926 02:58:34.009974 10381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:58:34 dist-test-slave-ktng krb5kdc[10153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.252@KRBTEST.COM for kudu/127.19.185.253@KRBTEST.COM
Sep 26 02:58:34 dist-test-slave-ktng krb5kdc[10153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:58:34.024217 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.010696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:39519 (local address 127.0.0.1:57196)
0926 02:58:34.010928 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:34.010945 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:34.011106 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:34.011501 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:34.011508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:34.011979 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:34.012188 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:34.012203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.013341 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.013345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:34.014578 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:34.014594 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.015366 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.015372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:34.015528 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.016112 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:34.016129 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:34.018190 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:34.021809 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:34.021814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:34.021818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:34.022169 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:34.022561 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:34.022565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:34.022569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:34.022709 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:34.023223 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:34.023228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:34.023419 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 02:58:34.023569 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 02:58:34.023844 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260926 02:58:34.024235 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.010803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.19.185.253:39519)
0926 02:58:34.010994 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.011001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.011021 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:58:34.011228 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.011234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.011351 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.011457 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.012408 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.013193 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.014750 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.015189 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.015349 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.018383 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:34.018388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:34.018391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:34.018422 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:34.021628 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.022297 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.022302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.022306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.022446 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.022817 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.022822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.022825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.023091 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:34.023254 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.023659 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.023872 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:58:34.025620 10381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:58:34.025761 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.011022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.19.185.254:41873)
0926 02:58:34.011182 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.011188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.011361 (+ 173us) server_negotiation.cc:407] Connection header received
0926 02:58:34.011567 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.011572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.011669 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.011744 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.013035 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.013843 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.015387 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.015821 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.015967 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.020396 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:34.020403 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:34.020410 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:34.020458 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:34.023140 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.023838 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.023848 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.023850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.023967 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.024340 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.024346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.024349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.024639 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:34.024942 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.025258 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.025484 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 02:58:34 dist-test-slave-ktng krb5kdc[10153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.252@KRBTEST.COM for kudu/127.19.185.252@KRBTEST.COM
Sep 26 02:58:34 dist-test-slave-ktng krb5kdc[10153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.253@KRBTEST.COM
I20260926 02:58:34.040913 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.026508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.19.185.252:36349)
0926 02:58:34.026720 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.026726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.027061 (+ 335us) server_negotiation.cc:407] Connection header received
0926 02:58:34.027122 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.027129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.027215 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.027365 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.028241 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.029088 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.030433 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.030988 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.031093 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.034658 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:34.034664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:34.034669 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:34.034710 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:34.038153 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.038801 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.038809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.038813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.038955 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.039371 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.039376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.039380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.039678 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:34.039851 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.040207 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.040659 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":217,"server-negotiator.queue_time_us":73}
I20260926 02:58:34.040958 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.026540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:42054)
0926 02:58:34.026792 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:34.026812 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:34.026914 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:34.027433 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:34.027437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:34.027956 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:34.028102 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:34.028110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.029241 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.029247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:34.030292 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:34.030304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.031138 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.031144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:34.031413 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.032143 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:34.032160 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:34.034465 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:34.038286 (+ 3821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:34.038293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:34.038297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:34.038677 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:34.039070 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:34.039078 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:34.039083 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:34.039226 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:34.039802 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:34.039807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:34.039961 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 02:58:34.040129 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 02:58:34.040497 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260926 02:58:34.043931 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.027858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.19.185.253:39519)
0926 02:58:34.028055 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.028062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.028082 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:58:34.028131 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.028136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.028234 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.028354 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.029725 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.030563 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.032086 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.032614 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.032717 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.037772 (+ 5055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:34.037777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:34.037781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:34.037812 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:34.040362 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.041264 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.041271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.041275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.041504 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.041984 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.041987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.041989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.042381 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:34.042740 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.043380 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.043681 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:58:34.048002 10381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d: Bootstrap starting.
I20260926 02:58:34.049294 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:34.052374 10381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:34.053519 10381 log.cc:824] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:34.053045 10208 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "67e44e9e35984f33a1c0fb44c27e8391"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58804
I20260926 02:58:34.056754 10381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d: No bootstrap required, opened a new log
I20260926 02:58:34.064656 10381 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } }
I20260926 02:58:34.065042 10381 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0775c0c2f504e8ea4a028a6b8ac948d, State: Initialized, Role: FOLLOWER
I20260926 02:58:34.065549 10381 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } }
I20260926 02:58:34.066275 10233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } attempt: 4
I20260926 02:58:34.066594 10388 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } } }
I20260926 02:58:34.066990 10388 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:58:34.068096 10381 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 02:58:34 dist-test-slave-ktng krb5kdc[10153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391512, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for kudu/127.19.185.252@KRBTEST.COM
W20260926 02:58:34.079433 10399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:58:34.079563 10399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:58:34.084587 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.067096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.19.185.252:36349)
0926 02:58:34.067255 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.067261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.067502 (+ 241us) server_negotiation.cc:407] Connection header received
0926 02:58:34.067728 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.067732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.067808 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.067903 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.069054 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.070260 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.071721 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.072577 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.072726 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.077395 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:34.077403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:34.077414 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:34.077454 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:34.081062 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.081954 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.081960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.081966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.082141 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.082625 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.082630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.082634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.083120 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:34.083280 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.083838 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.084195 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 02:58:34.084589 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.067039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:42066)
0926 02:58:34.067389 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:34.067412 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:34.067567 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:34.067974 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:34.067980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:34.068601 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:34.068883 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:34.068899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.070411 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.070416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:34.071580 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:34.071593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.072809 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.072819 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:34.073093 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.073858 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:34.073896 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:34.077140 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:34.081202 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:34.081211 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:34.081215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:34.081808 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:34.082279 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:34.082287 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:34.082291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:34.082492 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:34.083278 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:34.083285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:34.083507 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 02:58:34.083705 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 02:58:34.084112 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260926 02:58:34.091225 10233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391: Bootstrap starting.
I20260926 02:58:34.077087 10314 master_runner.cc:186] Error getting master registration for 127.19.185.254:41873: OK, instance_id { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" instance_seqno: 1790391512926827 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:58:34.092353 10314 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:58:34.095964 10233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:34.097466 10233 log.cc:824] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:34.100907 10233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391: No bootstrap required, opened a new log
I20260926 02:58:34.108232 10233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } }
I20260926 02:58:34.108592 10233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e44e9e35984f33a1c0fb44c27e8391, State: Initialized, Role: FOLLOWER
I20260926 02:58:34.109052 10233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } }
I20260926 02:58:34.111012 10233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:34.111155 10208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 02:58:34.111428 10208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:34.110378 10402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } } }
I20260926 02:58:34.115233 10208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } }
I20260926 02:58:34.115772 10402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:58:34.118166 10356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67e44e9e35984f33a1c0fb44c27e8391" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d"
I20260926 02:58:34.118788 10356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:34.122745 10356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e44e9e35984f33a1c0fb44c27e8391 in term 1.
I20260926 02:58:34.123572 10185 leader_election.cc:302] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e44e9e35984f33a1c0fb44c27e8391, a0775c0c2f504e8ea4a028a6b8ac948d; no voters:
I20260926 02:58:34.124182 10402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:34.124212 10208 leader_election.cc:288] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [CANDIDATE]: Term 1 election: Requested vote from peers 0c97679031a84d669bcccec539d9792f (127.19.185.253:39519), a0775c0c2f504e8ea4a028a6b8ac948d (127.19.185.252:36349)
I20260926 02:58:34.132097 10402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [term 1 LEADER]: Becoming Leader. State: Replica: 67e44e9e35984f33a1c0fb44c27e8391, State: Running, Role: LEADER
I20260926 02:58:34.133585 10402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } }
I20260926 02:58:34.150197 10409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67e44e9e35984f33a1c0fb44c27e8391. Latest consensus state: current_term: 1 leader_uuid: "67e44e9e35984f33a1c0fb44c27e8391" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } } }
I20260926 02:58:34.150848 10409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [sys.catalog]: This master's current role is: LEADER
W20260926 02:58:34.159107 10420 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:58:34.159350 10420 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 02:58:34.166097 10187 tablet.cc:2369] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:58:34.176923 10307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } attempt: 5
I20260926 02:58:34.159337 10166 master_runner.cc:186] Error getting master registration for 127.19.185.253:39519: OK, instance_id { permanent_uuid: "0c97679031a84d669bcccec539d9792f" instance_seqno: 1790391513449719 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:58:34.192129 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.160415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.19.185.254:41873)
0926 02:58:34.160645 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.160653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.161675 (+ 1022us) server_negotiation.cc:407] Connection header received
0926 02:58:34.162095 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.162101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.162178 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.162262 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.165990 (+ 3728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.167199 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.170677 (+ 3478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.171407 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.171504 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.178212 (+ 6708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:34.178218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:34.178223 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:34.178304 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:34.183098 (+ 4794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.184774 (+ 1676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.184780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.184784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.185043 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.185887 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.185895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.185898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.186606 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:34.187089 (+ 483us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.190719 (+ 3630us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.190990 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:58:34.192337 10166 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:58:34.192875 10418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:58:34.203274 10418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 02:58:34 dist-test-slave-ktng krb5kdc[10153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391513, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.253@KRBTEST.COM for kudu/127.19.185.252@KRBTEST.COM
W20260926 02:58:34.208333 10335 tablet.cc:2369] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:58:34.217890 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.169470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42074 (local address 127.19.185.252:36349)
0926 02:58:34.169680 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.169686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.176872 (+ 7186us) server_negotiation.cc:407] Connection header received
0926 02:58:34.177403 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.177412 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.177550 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.177648 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.185622 (+ 7974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.186845 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.192169 (+ 5324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.192977 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.193087 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.207089 (+ 14002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:34.207097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:34.207103 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:34.207150 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:34.211662 (+ 4512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.213323 (+ 1661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.213343 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.213350 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.213532 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.215245 (+ 1713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.215251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.215256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.215656 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:34.215948 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.217160 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.217462 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:58:34.218242 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.161933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57210 (local address 127.19.185.253:39519)
0926 02:58:34.162122 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.162128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.176046 (+ 13918us) server_negotiation.cc:407] Connection header received
0926 02:58:34.177960 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.177969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.178077 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.178170 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.182102 (+ 3932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.183368 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.188813 (+ 5445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.189854 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.190000 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.201790 (+ 11790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:34.201798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:34.201805 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:34.201860 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:34.206069 (+ 4209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.212591 (+ 6522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.212597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.212602 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.212776 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.214048 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.214054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.214058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.214471 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:34.214723 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.216544 (+ 1821us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.216845 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:58:34.222563 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.177508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:36349 (local address 127.0.0.1:42082)
0926 02:58:34.177883 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:34.177902 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:34.178009 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:34.197811 (+ 19802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:34.197819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:34.198365 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:34.198601 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:34.198618 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.201210 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.201217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:34.202739 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:34.202759 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.204924 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.204930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:34.205269 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.205904 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:34.205930 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:34.209937 (+ 4007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:34.216893 (+ 6956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:34.216898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:34.216902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:34.217400 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:34.220392 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:34.220397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:34.220401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:34.220600 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:34.221598 (+ 998us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:34.221604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:34.221762 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 02:58:34.221899 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 02:58:34.222176 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252}
I20260926 02:58:34.224789 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.178216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.19.185.252:36349)
0926 02:58:34.197451 (+ 19235us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.197456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.197481 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:58:34.197533 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.197538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.197620 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.197756 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.198800 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.200943 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.203020 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.204739 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.210123 (+ 5384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.210455 (+ 332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:34.210464 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:34.210469 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:34.210507 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:34.216685 (+ 6178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.219839 (+ 3154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.219879 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.219885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.220178 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:34.220798 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:34.220804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:34.220807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:34.221407 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:34.223275 (+ 1868us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.223583 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.223957 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19083,"thread_start_us":19725,"threads_started":1}
I20260926 02:58:34.232939 10356 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d [term 1 FOLLOWER]: Refusing update from remote peer 67e44e9e35984f33a1c0fb44c27e8391: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:58:34.233433 10307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f: Bootstrap starting.
I20260926 02:58:34.234171 10409 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:58:34.243180 10307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:34.245559 10307 log.cc:824] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:34.263243 10388 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67e44e9e35984f33a1c0fb44c27e8391. Latest consensus state: current_term: 1 leader_uuid: "67e44e9e35984f33a1c0fb44c27e8391" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } } }
I20260926 02:58:34.263648 10388 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:58:34.263903 10307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f: No bootstrap required, opened a new log
I20260926 02:58:34.271598 10427 mvcc.cc:205] Tried to move back new op lower bound from 7333443642283122688 to 7333443641905913856. Current Snapshot: MvccSnapshot[applied={T|T < 7333443642283122688}]
I20260926 02:58:34.282955 10402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67e44e9e35984f33a1c0fb44c27e8391" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } } }
I20260926 02:58:34.283468 10402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:34.284813 10307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } }
I20260926 02:58:34.285466 10307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c97679031a84d669bcccec539d9792f, State: Initialized, Role: FOLLOWER
I20260926 02:58:34.286237 10307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } }
I20260926 02:58:34.288722 10388 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67e44e9e35984f33a1c0fb44c27e8391" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } } }
I20260926 02:58:34.289001 10388 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:58:34.290745 10418 catalog_manager.cc:1347] Generated new cluster ID: ac8b121959094f379dd459aeb011e2a9
I20260926 02:58:34.290880 10418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:34.296849 10282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67e44e9e35984f33a1c0fb44c27e8391" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0c97679031a84d669bcccec539d9792f"
I20260926 02:58:34.297704 10307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:34.300344 10281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:34.297371 10432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } } }
I20260926 02:58:34.301407 10432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:58:34.304947 10281 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f [term 1 FOLLOWER]: Refusing update from remote peer 67e44e9e35984f33a1c0fb44c27e8391: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:58:34.308910 10282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e44e9e35984f33a1c0fb44c27e8391 in term 1.
I20260926 02:58:34.313112 10409 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:58:34.322585 10418 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:34.325310 10418 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:34.358546 10409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67e44e9e35984f33a1c0fb44c27e8391" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } } }
I20260926 02:58:34.358851 10409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:34.363497 10418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67e44e9e35984f33a1c0fb44c27e8391: Generated new TSK 0
I20260926 02:58:34.366204 10418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:34.381374 10432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67e44e9e35984f33a1c0fb44c27e8391. Latest consensus state: current_term: 1 leader_uuid: "67e44e9e35984f33a1c0fb44c27e8391" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } } }
I20260926 02:58:34.381734 10432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:58:34.384545 10440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67e44e9e35984f33a1c0fb44c27e8391" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e44e9e35984f33a1c0fb44c27e8391" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 41873 } } peers { permanent_uuid: "0c97679031a84d669bcccec539d9792f" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 39519 } } peers { permanent_uuid: "a0775c0c2f504e8ea4a028a6b8ac948d" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 36349 } } }
I20260926 02:58:34.384864 10440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:58:34.447190 10449 catalog_manager.cc:1259] Loaded cluster ID: ac8b121959094f379dd459aeb011e2a9
I20260926 02:58:34.447324 10449 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f: loading cluster ID for follower catalog manager: success
I20260926 02:58:34.452062 10449 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f: acquiring CA information for follower catalog manager: success
I20260926 02:58:34.456597 10449 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c97679031a84d669bcccec539d9792f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 02:58:34.525435 10385 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:34.512968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58828 (local address 127.19.185.254:41873)
0926 02:58:34.513202 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.513209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.522503 (+ 9294us) server_negotiation.cc:407] Connection header received
0926 02:58:34.523709 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.523716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.523832 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.523918 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.525156 (+ 1238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58828: BlockingRecv error: recv got EOF from 127.0.0.1:58828 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 02:58:34.530368 10311 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:34.527653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.19.185.253:39519)
0926 02:58:34.527841 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.527848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.528549 (+ 701us) server_negotiation.cc:407] Connection header received
0926 02:58:34.528819 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.528825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.528927 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.529001 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.530098 (+ 1097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57212: BlockingRecv error: recv got EOF from 127.0.0.1:57212 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 02:58:34.532976 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:34.516906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.19.185.252:36349)
0926 02:58:34.517106 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.517113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.527063 (+ 9950us) server_negotiation.cc:407] Connection header received
0926 02:58:34.530822 (+ 3759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.530829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.530943 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.531023 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.532740 (+ 1717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42092: BlockingRecv error: recv got EOF from 127.0.0.1:42092 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:58:34.596765 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.577404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57218 (local address 127.19.185.253:39519)
0926 02:58:34.577608 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.577615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.588267 (+ 10652us) server_negotiation.cc:407] Connection header received
0926 02:58:34.588848 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.588854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.589059 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.589207 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:34.590082 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.591347 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.593483 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.594195 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.594316 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.595118 (+ 802us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:34.595906 (+ 788us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:34.596026 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.596292 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.596401 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 02:58:34.642469 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:34.633084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.19.185.252:36349)
0926 02:58:34.633294 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.633301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.640218 (+ 6917us) server_negotiation.cc:407] Connection header received
0926 02:58:34.640788 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.640795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.640933 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.641012 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.642164 (+ 1152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42104: BlockingRecv error: recv got EOF from 127.0.0.1:42104 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:58:34.695747 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.677316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57220 (local address 127.19.185.253:39519)
0926 02:58:34.677510 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.677518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.688183 (+ 10665us) server_negotiation.cc:407] Connection header received
0926 02:58:34.688718 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.688725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.688853 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.688926 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:34.689612 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.690847 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.693336 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.694060 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.694164 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.694790 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:34.694939 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:34.695017 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.695274 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.695379 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 02:58:34.738569 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:34.729428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.19.185.252:36349)
0926 02:58:34.729644 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.729650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.736519 (+ 6869us) server_negotiation.cc:407] Connection header received
0926 02:58:34.737035 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.737043 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.737140 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.737220 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.738327 (+ 1107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42118: BlockingRecv error: recv got EOF from 127.0.0.1:42118 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:58:34.801542 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.783615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.19.185.253:39519)
0926 02:58:34.783809 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.783816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.794550 (+ 10734us) server_negotiation.cc:407] Connection header received
0926 02:58:34.795068 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.795075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.795175 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.795257 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:34.795933 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.797170 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.799344 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.800036 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.800177 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.800679 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:34.800830 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:34.800917 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.801091 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.801195 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 02:58:34.810925 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:34.808851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.19.185.252:36349)
0926 02:58:34.809055 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.809062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.809370 (+ 308us) server_negotiation.cc:407] Connection header received
0926 02:58:34.809543 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.809549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.809645 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.809750 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.810658 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42132: BlockingRecv error: recv got EOF from 127.0.0.1:42132 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:58:34.825278 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.818912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.19.185.253:39519)
0926 02:58:34.819132 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.819138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.819646 (+ 508us) server_negotiation.cc:407] Connection header received
0926 02:58:34.819838 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.819844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.819941 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.820060 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:34.820484 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.821665 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.823330 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.824008 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.824139 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.824509 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:34.824648 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:34.824745 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.824899 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.824983 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
W20260926 02:58:34.832934 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:34.831061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.19.185.252:36349)
0926 02:58:34.831357 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.831363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.831635 (+ 272us) server_negotiation.cc:407] Connection header received
0926 02:58:34.831797 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.831803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.831895 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.832004 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.832706 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42146: BlockingRecv error: recv got EOF from 127.0.0.1:42146 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
I20260926 02:58:34.850708 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.844462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.19.185.253:39519)
0926 02:58:34.844724 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.844730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.845103 (+ 373us) server_negotiation.cc:407] Connection header received
0926 02:58:34.845301 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.845305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.845370 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.845456 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:34.845952 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.846757 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.848771 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.849313 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.849424 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.849906 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:34.850016 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:34.850090 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.850278 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.850345 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 02:58:34.859601 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:34.857371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42150 (local address 127.19.185.252:36349)
0926 02:58:34.857556 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.857562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.858094 (+ 532us) server_negotiation.cc:407] Connection header received
0926 02:58:34.858267 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.858271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.858355 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.858450 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.859343 (+ 893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42150: BlockingRecv error: recv got EOF from 127.0.0.1:42150 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:58:34.886556 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.879768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.19.185.253:39519)
0926 02:58:34.880035 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.880042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.880613 (+ 571us) server_negotiation.cc:407] Connection header received
0926 02:58:34.880825 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.880832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.880931 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.881022 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:34.881583 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.882550 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.884597 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.885038 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.885319 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.885734 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:34.885851 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:34.885963 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.886110 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.886204 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
W20260926 02:58:34.897706 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:34.894981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42160 (local address 127.19.185.252:36349)
0926 02:58:34.895188 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.895196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.896006 (+ 810us) server_negotiation.cc:407] Connection header received
0926 02:58:34.896226 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.896232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.896319 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.896440 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.897435 (+ 995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42160: BlockingRecv error: recv got EOF from 127.0.0.1:42160 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:58:34.939888 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:34.933891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.19.185.253:39519)
0926 02:58:34.934146 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.934156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.934717 (+ 561us) server_negotiation.cc:407] Connection header received
0926 02:58:34.934959 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.934964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.935050 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.935157 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:34.935624 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.936615 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.937975 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:34.938463 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:34.938601 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:34.939006 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:34.939111 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:34.939219 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:34.939404 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 02:58:34.939480 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260926 02:58:34.951356 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:34.949415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42164 (local address 127.19.185.252:36349)
0926 02:58:34.949629 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:34.949635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:34.949924 (+ 289us) server_negotiation.cc:407] Connection header received
0926 02:58:34.950125 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:34.950130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:34.950217 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:34.950325 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:34.951107 (+ 782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42164: BlockingRecv error: recv got EOF from 127.0.0.1:42164 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 02:58:35.025527 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:35.019565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.19.185.253:39519)
0926 02:58:35.019767 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:35.019774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:35.020144 (+ 370us) server_negotiation.cc:407] Connection header received
0926 02:58:35.020318 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:35.020324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:35.020417 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:35.020539 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:35.020923 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:35.022052 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:35.023378 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:35.024173 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:35.024321 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:35.024673 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:35.024823 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:35.024917 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:35.025094 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 02:58:35.025182 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 02:58:35.033706 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:35.031791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.19.185.252:36349)
0926 02:58:35.031952 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:35.031956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:35.032381 (+ 425us) server_negotiation.cc:407] Connection header received
0926 02:58:35.032537 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:35.032541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:35.032611 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:35.032706 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:35.033475 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42170: BlockingRecv error: recv got EOF from 127.0.0.1:42170 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:58:35.081527 10399 catalog_manager.cc:1259] Loaded cluster ID: ac8b121959094f379dd459aeb011e2a9
I20260926 02:58:35.081665 10399 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d: loading cluster ID for follower catalog manager: success
I20260926 02:58:35.084127 10399 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d: acquiring CA information for follower catalog manager: success
I20260926 02:58:35.085870 10399 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a0775c0c2f504e8ea4a028a6b8ac948d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:58:35.176179 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:35.171082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.19.185.253:39519)
0926 02:58:35.171289 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:35.171296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:35.171567 (+ 271us) server_negotiation.cc:407] Connection header received
0926 02:58:35.171812 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:35.171816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:35.171892 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:35.171997 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:35.172410 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:35.173101 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:35.174463 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:35.174880 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:35.175024 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:35.175459 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:35.175575 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:35.175677 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:35.175822 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 02:58:35.175917 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:58:35.188095 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:35.182468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.19.185.252:36349)
0926 02:58:35.182630 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:35.182635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:35.182910 (+ 275us) server_negotiation.cc:407] Connection header received
0926 02:58:35.183050 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:35.183054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:35.183207 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:35.183377 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:35.183705 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:35.184416 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:35.185802 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:35.186229 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:35.186332 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:35.186871 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:35.187522 (+ 651us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:35.187642 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:35.187801 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 02:58:35.187869 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 02:58:35.204993 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:35.194653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.19.185.252:36349)
0926 02:58:35.194871 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:35.194893 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:35.198607 (+ 3714us) server_negotiation.cc:407] Connection header received
0926 02:58:35.198787 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:35.198793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:35.198899 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:35.198989 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:35.200022 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:35.200860 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:35.202714 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:35.203417 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:35.203559 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:35.204151 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:35.204275 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:35.204335 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:35.204626 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 02:58:35.204706 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:58:35.205214 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:35.194458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.19.185.254:41873)
0926 02:58:35.198284 (+ 3826us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:35.198290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:35.198313 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:58:35.198444 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:35.198449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:35.198678 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:35.198852 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:35.199597 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:35.200707 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:35.202909 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:35.203344 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:35.203564 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:35.204055 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:35.204401 (+ 346us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:35.204519 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:35.204840 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 02:58:35.204919 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3679,"thread_start_us":3551,"threads_started":1}
I20260926 02:58:35.207165 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:35.199238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.19.185.253:39519)
0926 02:58:35.199454 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:35.199460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:35.200161 (+ 701us) server_negotiation.cc:407] Connection header received
0926 02:58:35.200448 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:35.200455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:35.200533 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:35.200590 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:35.201558 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:35.202680 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:35.204722 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:35.205447 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:35.205519 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:35.206436 (+ 917us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:35.206545 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:35.206594 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:35.206825 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 02:58:35.206886 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 02:58:35.210821 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 10166
I20260926 02:58:35.211879 10228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:58:35.323815 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 10166
I20260926 02:58:35.345094 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 10240
I20260926 02:58:35.346233 10302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:58:35.453627 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 10240
I20260926 02:58:35.473790 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 10314
I20260926 02:58:35.474900 10376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:58:35.581348 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 10314
2026-09-26T02:58:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4489 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:58:35 dist-test-slave-ktng krb5kdc[10563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:58:35 dist-test-slave-ktng krb5kdc[10563](info): set up 2 sockets
Sep 26 02:58:35 dist-test-slave-ktng krb5kdc[10563](info): commencing operation
krb5kdc: starting...
W20260926 02:58:36.756848 20199 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/test-user.keytab.
Sep 26 02:58:36 dist-test-slave-ktng krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:58:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.185.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
Entry for principal kudu/127.19.185.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab.
I20260926 02:58:37.031862 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:38379
--webserver_interface=127.19.185.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:38379,127.19.185.253:33491,127.19.185.252:43353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:58:37.368407 10576 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:58:37.368736 10576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:37.368798 10576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:37.368873 10576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:37.376242 10576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:37.376338 10576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:37.376430 10576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:37.376511 10576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:37.376621 10576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:37.386933 10576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:38379,127.19.185.253:33491,127.19.185.252:43353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.254.keytab
--principal=kudu/127.19.185.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.254:38379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.185.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:37.388412 10576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:37.389771 10576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:37.402658 10581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:37.402711 10582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:37.402688 10584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:37.403312 10576 server_base.cc:1034] running on GCE node
I20260926 02:58:37.404157 10576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:37.405632 10576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:37.406875 10576 hybrid_clock.cc:648] HybridClock initialized: now 1790391517406813 us; error 36 us; skew 500 ppm
I20260926 02:58:37.407228 10576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:37 dist-test-slave-ktng krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391517, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:37.410501 10576 init.cc:377] Logged in from keytab as kudu/127.19.185.254@KRBTEST.COM (short username kudu)
I20260926 02:58:37.412086 10576 webserver.cc:460] Webserver started at http://127.19.185.254:44671/ using document root <none> and password file <none>
I20260926 02:58:37.412801 10576 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:37.412890 10576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:37.413223 10576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:37.416296 10576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/data/instance:
uuid: "fb7e7ee48b3747998866814cce9be049"
format_stamp: "Formatted at 2026-09-26 02:58:37 on dist-test-slave-ktng"
server_key: "a64eff75e1132b718cb92ecd5e8c5950"
server_key_iv: "e040bc243946fa84cd493f924c190e14"
server_key_version: "encryptionkey@0"
I20260926 02:58:37.417101 10576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/wal/instance:
uuid: "fb7e7ee48b3747998866814cce9be049"
format_stamp: "Formatted at 2026-09-26 02:58:37 on dist-test-slave-ktng"
server_key: "a64eff75e1132b718cb92ecd5e8c5950"
server_key_iv: "e040bc243946fa84cd493f924c190e14"
server_key_version: "encryptionkey@0"
I20260926 02:58:37.422310 10576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:58:37.425892 10591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:37.427495 10576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:58:37.427686 10576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/wal
uuid: "fb7e7ee48b3747998866814cce9be049"
format_stamp: "Formatted at 2026-09-26 02:58:37 on dist-test-slave-ktng"
server_key: "a64eff75e1132b718cb92ecd5e8c5950"
server_key_iv: "e040bc243946fa84cd493f924c190e14"
server_key_version: "encryptionkey@0"
I20260926 02:58:37.427919 10576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:37.450142 10576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:37.455482 10576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:37.455833 10576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:37.476876 10576 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.254:38379
I20260926 02:58:37.476902 10642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.254:38379 every 8 connection(s)
I20260926 02:58:37.478451 10576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/data/info.pb
I20260926 02:58:37.482412 10643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:37.488859 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 10576
I20260926 02:58:37.489097 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-0/wal/instance
I20260926 02:58:37.489598 20199 external_mini_cluster.cc:1428] Setting key 8c64d55fcb39015ba69304e774a6737a
I20260926 02:58:37.489418 10643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:58:37 dist-test-slave-ktng krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391517, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:58:37.516286 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:37.492692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38379 (local address 127.0.0.1:49128)
0926 02:58:37.493566 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:37.493614 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:37.495889 (+ 2275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:37.496973 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:37.496979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:37.497595 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:37.498044 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:37.498062 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:37.500184 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:37.500193 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:37.502223 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:37.502244 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:37.503426 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:37.503436 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:37.503702 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:37.504483 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:37.504518 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:37.507860 (+ 3342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:37.511371 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:37.511392 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:37.511409 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:37.512054 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:37.512522 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:37.512530 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:37.512536 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:37.512771 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:37.513871 (+ 1100us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:37.513883 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:37.514172 (+ 289us) client_negotiation.cc:769] Sending connection context
0926 02:58:37.514496 (+ 324us) client_negotiation.cc:240] Negotiation successful
0926 02:58:37.515271 (+ 775us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":92,"thread_start_us":387,"threads_started":1}
I20260926 02:58:37.516358 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:37.494056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.19.185.254:38379)
0926 02:58:37.494821 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:37.494833 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:37.494896 (+ 63us) server_negotiation.cc:407] Connection header received
0926 02:58:37.496322 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:37.496333 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:37.496689 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:37.496923 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:37.498282 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:37.500009 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:37.502409 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:37.503097 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:37.503245 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:37.508127 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:37.508136 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:37.508149 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:37.508195 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:37.511181 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:37.512213 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:37.512220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:37.512223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:37.512387 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:37.512944 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:37.512949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:37.512951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:37.513701 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:37.513886 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:37.514664 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 02:58:37.514990 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":529,"thread_start_us":349,"threads_started":1}
WARNING: no policy specified for kudu/127.19.185.253@KRBTEST.COM; defaulting to no policy
I20260926 02:58:37.520138 10643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.253@KRBTEST.COM" created.
I20260926 02:58:37.521508 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:37.520962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:33491 (local address 127.0.0.1:44558)
0926 02:58:37.521149 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:37.521291 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 02:58:37.521904 10592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
I20260926 02:58:37.523756 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:37.523354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:33491 (local address 127.0.0.1:44560)
0926 02:58:37.523530 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:37.523605 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 02:58:37.524091 10643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:33491: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.253.keytab.
Entry for principal kudu/127.19.185.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.253.keytab.
I20260926 02:58:37.556406 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.253:33491
--webserver_interface=127.19.185.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.253.keytab
--principal=kudu/127.19.185.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:38379,127.19.185.253:33491,127.19.185.252:43353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:58:37.573518 10643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } attempt: 1
I20260926 02:58:37.574958 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:37.574354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:33491 (local address 127.0.0.1:44572)
0926 02:58:37.574619 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:37.574773 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 02:58:37.576236 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:37.575744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:33491 (local address 127.0.0.1:44582)
0926 02:58:37.575936 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:37.576040 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 02:58:37.576471 10643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:33491: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
I20260926 02:58:37.644783 10643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } attempt: 2
I20260926 02:58:37.646131 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:37.645571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:33491 (local address 127.0.0.1:44584)
0926 02:58:37.645850 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:37.645962 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 02:58:37.647411 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:37.646935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:33491 (local address 127.0.0.1:44598)
0926 02:58:37.647113 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:37.647205 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 02:58:37.647665 10643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:33491: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
I20260926 02:58:37.738952 10643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } attempt: 3
I20260926 02:58:37.740459 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:37.739840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:33491 (local address 127.0.0.1:44600)
0926 02:58:37.740114 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:37.740261 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 02:58:37.741686 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:37.741276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:33491 (local address 127.0.0.1:44604)
0926 02:58:37.741442 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:37.741526 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 02:58:37.741978 10643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:33491: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
I20260926 02:58:37.885263 10643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } attempt: 4
I20260926 02:58:37.886536 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:37.885994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:33491 (local address 127.0.0.1:44616)
0926 02:58:37.886307 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:37.886400 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260926 02:58:37.887419 10650 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260926 02:58:37.887797 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:37.887345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:33491 (local address 127.0.0.1:44630)
0926 02:58:37.887531 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:37.887637 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 02:58:37.887912 10650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:37.888021 10650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:37.888032 10643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.253:33491: Network error: Client connection negotiation failed: client connection to 127.19.185.253:33491: connect: Connection refused (error 111)
W20260926 02:58:37.888193 10650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:37.896713 10650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:37.896848 10650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:37.896970 10650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:37.897054 10650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:37.897146 10650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:37.907222 10650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:38379,127.19.185.253:33491,127.19.185.252:43353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.253.keytab
--principal=kudu/127.19.185.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.253:33491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.185.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:37.908572 10650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:37.909915 10650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:37.921826 10658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:37.921852 10655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:37.921887 10656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:37.922396 10650 server_base.cc:1034] running on GCE node
I20260926 02:58:37.923182 10650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:37.924634 10650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:37.925872 10650 hybrid_clock.cc:648] HybridClock initialized: now 1790391517925801 us; error 50 us; skew 500 ppm
I20260926 02:58:37.926195 10650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:37 dist-test-slave-ktng krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391517, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:37.941282 10650 init.cc:377] Logged in from keytab as kudu/127.19.185.253@KRBTEST.COM (short username kudu)
I20260926 02:58:37.943044 10650 webserver.cc:460] Webserver started at http://127.19.185.253:46229/ using document root <none> and password file <none>
I20260926 02:58:37.943856 10650 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:37.944001 10650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:37.944360 10650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:37.947692 10650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/data/instance:
uuid: "a6efcfdd227e43f4874881244167d8e3"
format_stamp: "Formatted at 2026-09-26 02:58:37 on dist-test-slave-ktng"
server_key: "2ef5385ba5ef4d227c1162d021899f32"
server_key_iv: "b1115e85a103bc728c99016d91e11a66"
server_key_version: "encryptionkey@0"
I20260926 02:58:37.948782 10650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/wal/instance:
uuid: "a6efcfdd227e43f4874881244167d8e3"
format_stamp: "Formatted at 2026-09-26 02:58:37 on dist-test-slave-ktng"
server_key: "2ef5385ba5ef4d227c1162d021899f32"
server_key_iv: "b1115e85a103bc728c99016d91e11a66"
server_key_version: "encryptionkey@0"
I20260926 02:58:37.954238 10650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:58:37.957947 10665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:37.959450 10650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:58:37.959625 10650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/wal
uuid: "a6efcfdd227e43f4874881244167d8e3"
format_stamp: "Formatted at 2026-09-26 02:58:37 on dist-test-slave-ktng"
server_key: "2ef5385ba5ef4d227c1162d021899f32"
server_key_iv: "b1115e85a103bc728c99016d91e11a66"
server_key_version: "encryptionkey@0"
I20260926 02:58:37.959848 10650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:37.987185 10650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:37.991461 10650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:37.991750 10650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:38.010530 10650 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.253:33491
I20260926 02:58:38.010550 10716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.253:33491 every 8 connection(s)
I20260926 02:58:38.011979 10650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/data/info.pb
I20260926 02:58:38.013981 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 10650
I20260926 02:58:38.014186 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-1/wal/instance
I20260926 02:58:38.014751 20199 external_mini_cluster.cc:1428] Setting key 04df12718fc56708563b48fa0ba3b518
I20260926 02:58:38.016603 10717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:38.026736 10717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:58:38 dist-test-slave-ktng krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391517, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.253@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.185.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.185.252@KRBTEST.COM" created.
I20260926 02:58:38.048008 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.029517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.19.185.254:38379)
0926 02:58:38.030068 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:38.030073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:38.031014 (+ 941us) server_negotiation.cc:407] Connection header received
0926 02:58:38.033127 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:38.033133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:38.033229 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:38.033349 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:38.034956 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.035785 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.037338 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.037772 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.037923 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.041624 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:38.041629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:38.041632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:38.041668 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:38.044645 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.045587 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.045592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.045594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.045721 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.046205 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.046208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.046210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.046497 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:38.046609 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:38.047572 (+ 963us) server_negotiation.cc:299] Negotiation successful
0926 02:58:38.047737 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":317,"threads_started":1}
I20260926 02:58:38.049590 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.030119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38379 (local address 127.0.0.1:49144)
0926 02:58:38.030835 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.030887 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:38.032845 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:38.033604 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:38.033612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:38.034147 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:38.034722 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.034739 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.035967 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.035971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.037183 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.037196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.038030 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.038037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.038269 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.039074 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:38.039102 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:38.041389 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:38.044806 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.044816 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.044837 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.045434 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.045841 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.045847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.045851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.046093 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.046631 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:38.046643 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:38.047147 (+ 504us) client_negotiation.cc:769] Sending connection context
0926 02:58:38.047520 (+ 373us) client_negotiation.cc:240] Negotiation successful
0926 02:58:38.048112 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":293,"threads_started":1}
I20260926 02:58:38.051468 10717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:58:38 dist-test-slave-ktng krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391517, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.253@KRBTEST.COM for kudu/127.19.185.253@KRBTEST.COM
I20260926 02:58:38.068778 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.052132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:33491 (local address 127.0.0.1:44636)
0926 02:58:38.052424 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.052444 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:38.052548 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:38.053664 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:38.053667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:38.054027 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:38.054163 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.054171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.056143 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.056150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.057351 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.057363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.058649 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.058653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.058814 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.059435 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:38.059451 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:38.061422 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:38.066158 (+ 4736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.066162 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.066165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.066632 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.067071 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.067075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.067077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.067220 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.067989 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:38.067993 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:38.068125 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 02:58:38.068248 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 02:58:38.068484 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":3}
I20260926 02:58:38.069264 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.052424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.19.185.253:33491)
0926 02:58:38.053135 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:38.053145 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:38.053178 (+ 33us) server_negotiation.cc:407] Connection header received
0926 02:58:38.053236 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:38.053246 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:38.053502 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:38.053661 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:38.054329 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.056001 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.057503 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.058512 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.058655 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.061651 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:38.061667 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:38.061689 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:38.061731 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:38.065996 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.066781 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.066792 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.066799 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.066954 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.067357 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.067364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.067370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.067847 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:38.068080 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:38.068427 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 02:58:38.068717 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":363,"threads_started":1}
I20260926 02:58:38.072055 10717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:58:38.073192 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.072741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57746)
0926 02:58:38.072957 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.073054 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 02:58:38.073496 10666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
I20260926 02:58:38.075001 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.074684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57762)
0926 02:58:38.074837 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.074911 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 02:58:38.075249 10717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:43353: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.185.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.252.keytab.
Entry for principal kudu/127.19.185.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.252.keytab.
I20260926 02:58:38.083838 20199 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
/tmp/dist-test-taskAXkdvX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.252:43353
--webserver_interface=127.19.185.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.252.keytab
--principal=kudu/127.19.185.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.185.212:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.185.254:38379,127.19.185.253:33491,127.19.185.252:43353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:58:38.124611 10717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } attempt: 1
I20260926 02:58:38.125801 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.125356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57778)
0926 02:58:38.125545 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.125646 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 02:58:38.126809 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.126453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57784)
0926 02:58:38.126619 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.126688 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 02:58:38.127003 10717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:43353: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
I20260926 02:58:38.187279 10643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } attempt: 5
Sep 26 02:58:38 dist-test-slave-ktng krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391517, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for kudu/127.19.185.253@KRBTEST.COM
I20260926 02:58:38.195207 10717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } attempt: 2
I20260926 02:58:38.196307 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.195839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57788)
0926 02:58:38.196056 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.196147 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 02:58:38.197373 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.196982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57796)
0926 02:58:38.197185 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.197253 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260926 02:58:38.197556 10717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:43353: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
I20260926 02:58:38.202013 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.188207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:33491 (local address 127.0.0.1:44642)
0926 02:58:38.188441 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.188463 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:38.188593 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:38.188989 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:38.188992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:38.189376 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:38.189547 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.189556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.190990 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.190994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.191929 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.191943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.192960 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.192964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.193177 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.193818 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:38.193836 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:38.195942 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:38.199851 (+ 3909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.199855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.199858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.200214 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.200611 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.200615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.200617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.200739 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.201313 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:38.201317 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:38.201435 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 02:58:38.201550 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 02:58:38.201771 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 02:58:38.202499 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.188418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.19.185.253:33491)
0926 02:58:38.188635 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:38.188641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:38.188664 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:58:38.188731 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:38.188736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:38.188836 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:38.188948 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:38.189711 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.190842 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.192088 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.192745 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.192904 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.196120 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:38.196127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:38.196131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:38.196169 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:38.199709 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.200332 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.200338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.200341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.200503 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.200841 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.200845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.200848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.201198 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:38.201340 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:38.201697 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 02:58:38.201976 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:58:38.203567 10643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:58:38.204536 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.204179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57808)
0926 02:58:38.204329 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.204413 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 02:58:38.205468 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.205066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57812)
0926 02:58:38.205207 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.205332 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 02:58:38.205664 10643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:43353: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
I20260926 02:58:38.256910 10643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } attempt: 1
I20260926 02:58:38.258165 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.257610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57828)
0926 02:58:38.257846 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.257960 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 02:58:38.259207 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.258839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57832)
0926 02:58:38.258994 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.259080 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 02:58:38.259460 10643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:43353: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
I20260926 02:58:38.288774 10717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } attempt: 3
I20260926 02:58:38.289906 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.289440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57838)
0926 02:58:38.289637 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.289732 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 02:58:38.290912 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.290530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57846)
0926 02:58:38.290758 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.290818 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260926 02:58:38.291098 10717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:43353: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
I20260926 02:58:38.327698 10643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } attempt: 2
I20260926 02:58:38.329017 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.328416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57862)
0926 02:58:38.328686 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.328813 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 02:58:38.330138 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.329731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57874)
0926 02:58:38.329894 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.329982 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 02:58:38.330423 10643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:43353: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
W20260926 02:58:38.419267 10725 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:58:38.419606 10725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:58:38.419684 10725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:58:38.419766 10725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:58:38.426949 10725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:58:38.427043 10725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:58:38.427101 10725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:58:38.427152 10725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:58:38.427207 10725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:58:38.434284 10717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } attempt: 4
I20260926 02:58:38.435494 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.434896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57890)
0926 02:58:38.435155 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.435278 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 02:58:38.436569 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.436173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57904)
0926 02:58:38.436340 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.436436 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 02:58:38.436667 10643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } attempt: 3
W20260926 02:58:38.436863 10717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:43353: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
I20260926 02:58:38.437755 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.437245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57910)
0926 02:58:38.437444 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.437539 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 02:58:38.438542 10725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.185.212:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.185.254:38379,127.19.185.253:33491,127.19.185.252:43353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/krb5kdc/kudu_127.19.185.252.keytab
--principal=kudu/127.19.185.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.185.252:43353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.185.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:58:38.438827 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.438455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57914)
0926 02:58:38.438622 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.438712 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 02:58:38.439066 10643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.185.252:43353: Network error: Client connection negotiation failed: client connection to 127.19.185.252:43353: connect: Connection refused (error 111)
I20260926 02:58:38.440184 10725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:58:38.441479 10725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:58:38.453392 10730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:38.453529 10731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:58:38.453845 10733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:58:38.454156 10725 server_base.cc:1034] running on GCE node
I20260926 02:58:38.454906 10725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:58:38.456319 10725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:58:38.457571 10725 hybrid_clock.cc:648] HybridClock initialized: now 1790391518457509 us; error 38 us; skew 500 ppm
I20260926 02:58:38.457906 10725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:58:38 dist-test-slave-ktng krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391518, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:58:38.461041 10725 init.cc:377] Logged in from keytab as kudu/127.19.185.252@KRBTEST.COM (short username kudu)
I20260926 02:58:38.462515 10725 webserver.cc:460] Webserver started at http://127.19.185.252:39459/ using document root <none> and password file <none>
I20260926 02:58:38.463260 10725 fs_manager.cc:359] Metadata directory not provided
I20260926 02:58:38.463474 10725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:58:38.463799 10725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:58:38.466773 10725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/data/instance:
uuid: "e33462cca1804ea7b89c43e2ec41427a"
format_stamp: "Formatted at 2026-09-26 02:58:38 on dist-test-slave-ktng"
server_key: "d636a8135ffcbfd5998d9e04035e04d0"
server_key_iv: "8b75573d1d5597ce0b2de1b01d4e4dc3"
server_key_version: "encryptionkey@0"
I20260926 02:58:38.467633 10725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/wal/instance:
uuid: "e33462cca1804ea7b89c43e2ec41427a"
format_stamp: "Formatted at 2026-09-26 02:58:38 on dist-test-slave-ktng"
server_key: "d636a8135ffcbfd5998d9e04035e04d0"
server_key_iv: "8b75573d1d5597ce0b2de1b01d4e4dc3"
server_key_version: "encryptionkey@0"
I20260926 02:58:38.472414 10725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:58:38.475651 10740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:58:38.477231 10725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 02:58:38.477423 10725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/data,/tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/wal
uuid: "e33462cca1804ea7b89c43e2ec41427a"
format_stamp: "Formatted at 2026-09-26 02:58:38 on dist-test-slave-ktng"
server_key: "d636a8135ffcbfd5998d9e04035e04d0"
server_key_iv: "8b75573d1d5597ce0b2de1b01d4e4dc3"
server_key_version: "encryptionkey@0"
I20260926 02:58:38.477633 10725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:58:38.499651 10725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:58:38.504004 10725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:58:38.504225 10725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:58:38.522614 10725 rpc_server.cc:307] RPC server started. Bound to: 127.19.185.252:43353
I20260926 02:58:38.522632 10791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.185.252:43353 every 8 connection(s)
I20260926 02:58:38.524017 10725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/data/info.pb
I20260926 02:58:38.527844 10792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:58:38.528666 20199 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu as pid 10725
I20260926 02:58:38.528875 20199 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAXkdvX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391151502216-20199-0/minicluster-data/master-2/wal/instance
I20260926 02:58:38.529435 20199 external_mini_cluster.cc:1428] Setting key fc1c823975d695ffb3a7b42e29742efa
I20260926 02:58:38.537444 10792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:58:38 dist-test-slave-ktng krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.252@KRBTEST.COM
Sep 26 02:58:38 dist-test-slave-ktng krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391518, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.252@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:58:38.552860 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.531360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.19.185.252:43353)
0926 02:58:38.532196 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:38.532207 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:38.532296 (+ 89us) server_negotiation.cc:407] Connection header received
0926 02:58:38.534317 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:38.534339 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:38.534744 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:38.535104 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:38.536892 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.538188 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.539577 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.540545 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.540719 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.545232 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:38.545240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:38.545251 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:38.545291 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:38.548664 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.549332 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.549338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.549343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.549554 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.550005 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.550009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.550011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.550619 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:38.550861 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:38.551367 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 02:58:38.551762 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":371,"threads_started":1}
I20260926 02:58:38.554595 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.539955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.19.185.254:38379)
0926 02:58:38.540174 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:38.540183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:38.540921 (+ 738us) server_negotiation.cc:407] Connection header received
0926 02:58:38.541156 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:38.541168 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:38.541305 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:38.541407 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:38.542922 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.543709 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.545072 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.545485 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.545613 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.548964 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:38.548968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:38.548972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:38.549003 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:38.552010 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.552683 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.552690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.552693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.552823 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.553185 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.553193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.553196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.553453 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:38.553606 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:38.554048 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 02:58:38.554259 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:58:38.554720 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.539956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.254:38379 (local address 127.0.0.1:49152)
0926 02:58:38.540790 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.540832 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:38.541032 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:38.541523 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:38.541530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:38.542415 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:38.542767 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.542781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.543853 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.543858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.544922 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.544934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.545676 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.545686 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.545856 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.546405 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:38.546422 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:38.548763 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:38.552139 (+ 3376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.552151 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.552158 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.552571 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.552938 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.552944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.552947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.553090 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.553583 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:38.553591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:38.553767 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 02:58:38.553946 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 02:58:38.554255 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":435,"threads_started":1}
I20260926 02:58:38.556191 10792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:58:38 dist-test-slave-ktng krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391518, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.252@KRBTEST.COM for kudu/127.19.185.253@KRBTEST.COM
Sep 26 02:58:38 dist-test-slave-ktng krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.254@KRBTEST.COM
I20260926 02:58:38.569162 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.556779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.253:33491 (local address 127.0.0.1:44648)
0926 02:58:38.556910 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.556926 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:38.557024 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:38.557407 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:38.557412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:38.557767 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:38.557895 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.557902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.558798 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.558805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.559730 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.559741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.560434 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.560440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.560604 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.561232 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:38.561247 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:38.563595 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:38.566633 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.566638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.566641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.567149 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.567548 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.567555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.567559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.567730 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.568220 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:38.568225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:38.568395 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 02:58:38.568538 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 02:58:38.568807 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 02:58:38.569193 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.556953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.19.185.253:33491)
0926 02:58:38.557149 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:38.557161 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:38.557176 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:58:38.557223 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:38.557226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:38.557281 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:38.557385 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:38.558027 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.558664 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.559864 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.560278 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.560409 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.563843 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:38.563855 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:38.563859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:38.563913 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:38.566521 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.567257 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.567260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.567262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.567431 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.567834 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.567837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.567839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.568088 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:38.568251 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:38.568625 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 02:58:38.568820 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:58:38.570127 10792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:58:38.570803 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.556024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.19.185.254:38379)
0926 02:58:38.556200 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:38.556206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:38.556224 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:58:38.556325 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:38.556330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:38.556412 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:38.556531 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:38.557802 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.558471 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.559712 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.560132 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.560265 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.564901 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:38.564907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:38.564911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:38.564950 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:38.568073 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.568681 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.568687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.568691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.568831 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.569204 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.569210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.569213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.569582 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:38.569916 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:38.570275 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 02:58:38.570474 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 02:58:38 dist-test-slave-ktng krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391518, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.252@KRBTEST.COM for kudu/127.19.185.252@KRBTEST.COM
Sep 26 02:58:38 dist-test-slave-ktng krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.185.253@KRBTEST.COM
I20260926 02:58:38.583869 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.570859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57928)
0926 02:58:38.571212 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.571225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:38.571346 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:38.571677 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:38.571680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:38.572079 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:38.572224 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.572235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.573589 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.573593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.574528 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.574541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.575242 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.575250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.575440 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.575984 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:38.575999 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:38.578101 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:38.581577 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.581580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.581583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.581894 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.582330 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.582335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.582337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.582495 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.583007 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:38.583012 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:38.583160 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 02:58:38.583388 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 02:58:38.583623 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275}
I20260926 02:58:38.584553 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.571085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.19.185.252:43353)
0926 02:58:38.571255 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:38.571260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:38.571277 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:58:38.571460 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:38.571465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:38.571541 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:38.571643 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:38.572392 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.573453 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.574663 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.575077 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.575174 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.578267 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:38.578274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:38.578278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:38.578329 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:38.581407 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.582012 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.582018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.582022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.582172 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.582611 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.582619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.582623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.582893 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:38.583040 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:38.584193 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0926 02:58:38.584355 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:58:38.588263 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.572682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.19.185.253:33491)
0926 02:58:38.572828 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:38.572831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:38.572846 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:58:38.572926 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:38.572929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:38.573001 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:38.573071 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:38.574372 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.575178 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.576570 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.577198 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.577289 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.582507 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:38.582511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:38.582514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:38.582544 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:38.585503 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.586293 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.586298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.586301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.586510 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.586853 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.586858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.586861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.587194 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:38.587508 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:38.587709 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 02:58:38.587941 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 02:58:38.590281 10792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a: Bootstrap starting.
I20260926 02:58:38.593161 20199 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:58:38.593676 10792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:38.594688 10792 log.cc:824] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:38.597783 10792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a: No bootstrap required, opened a new log
I20260926 02:58:38.595520 10618 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fb7e7ee48b3747998866814cce9be049"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49164
I20260926 02:58:38.608572 10792 raft_consensus.cc:348] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } }
I20260926 02:58:38.609004 10792 raft_consensus.cc:729] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e33462cca1804ea7b89c43e2ec41427a, State: Initialized, Role: FOLLOWER
I20260926 02:58:38.609527 10792 consensus_queue.cc:260] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } }
I20260926 02:58:38.610433 10798 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } } }
I20260926 02:58:38.610750 10798 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:58:38.611744 10792 sys_catalog.cc:564] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:38.616298 10643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } attempt: 4
W20260926 02:58:38.621523 10809 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:58:38.621661 10809 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 26 02:58:38 dist-test-slave-ktng krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391517, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.254@KRBTEST.COM for kudu/127.19.185.252@KRBTEST.COM
I20260926 02:58:38.619108 10725 master_runner.cc:186] Error getting master registration for 127.19.185.254:38379: OK, instance_id { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" instance_seqno: 1790391517470764 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:58:38.631189 10725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:58:38.635175 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.617009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57938)
0926 02:58:38.617386 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.617407 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:38.617548 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:38.617950 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:38.617955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:38.618549 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:38.618770 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.618782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.620410 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.620415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.621873 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.621889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.622937 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.622943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.623143 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.623883 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:38.623908 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:38.627942 (+ 4034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:38.632343 (+ 4401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.632352 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.632357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.632886 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.633301 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.633306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.633309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.633469 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.634144 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:38.634148 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:38.634332 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 02:58:38.634487 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 02:58:38.634769 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260926 02:58:38.635214 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.617018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.19.185.252:43353)
0926 02:58:38.617212 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:38.617219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:38.617503 (+ 284us) server_negotiation.cc:407] Connection header received
0926 02:58:38.617685 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:38.617691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:38.617779 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:38.617905 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:38.618940 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.620194 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.622032 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.622772 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.622917 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.628180 (+ 5263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:38.628188 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:38.628193 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:38.628230 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:38.632198 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.633018 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.633024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.633028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.633175 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.633584 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.633590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.633593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.634026 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:38.634179 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:38.634632 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 02:58:38.634863 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:58:38.639984 10643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049: Bootstrap starting.
I20260926 02:58:38.643078 10643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:38.644169 10643 log.cc:824] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:38.646903 10643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049: No bootstrap required, opened a new log
I20260926 02:58:38.652383 10643 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } }
I20260926 02:58:38.652763 10643 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb7e7ee48b3747998866814cce9be049, State: Initialized, Role: FOLLOWER
I20260926 02:58:38.653205 10643 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } }
I20260926 02:58:38.654189 10812 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } } }
I20260926 02:58:38.654618 10812 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:58:38.655417 10643 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:38.655475 10618 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 02:58:38.655650 10618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:38.659770 10618 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } }
I20260926 02:58:38.662355 10618 leader_election.cc:288] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [CANDIDATE]: Term 1 election: Requested vote from peers a6efcfdd227e43f4874881244167d8e3 (127.19.185.253:33491), e33462cca1804ea7b89c43e2ec41427a (127.19.185.252:43353)
I20260926 02:58:38.663412 10767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb7e7ee48b3747998866814cce9be049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e33462cca1804ea7b89c43e2ec41427a"
I20260926 02:58:38.663920 10767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:38.668215 10767 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb7e7ee48b3747998866814cce9be049 in term 1.
I20260926 02:58:38.669101 10592 leader_election.cc:302] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e33462cca1804ea7b89c43e2ec41427a, fb7e7ee48b3747998866814cce9be049; no voters:
I20260926 02:58:38.669710 10812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:58:38.678516 10812 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [term 1 LEADER]: Becoming Leader. State: Replica: fb7e7ee48b3747998866814cce9be049, State: Running, Role: LEADER
I20260926 02:58:38.679628 10812 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } }
I20260926 02:58:38.692807 10819 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb7e7ee48b3747998866814cce9be049. Latest consensus state: current_term: 1 leader_uuid: "fb7e7ee48b3747998866814cce9be049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } } }
I20260926 02:58:38.693276 10819 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:38.703907 10826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260926 02:58:38.709087 10597 tablet.cc:2369] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:58:38.726117 10826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:58:38.736064 10717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } attempt: 5
I20260926 02:58:38.744890 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.703568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.19.185.254:38379)
0926 02:58:38.703771 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:38.703778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:38.707558 (+ 3780us) server_negotiation.cc:407] Connection header received
0926 02:58:38.708604 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:38.708614 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:38.708728 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:38.708832 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:38.714735 (+ 5903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.716263 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.720336 (+ 4073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.721146 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.721289 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.730308 (+ 9019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:38.730315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:38.730319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:38.730359 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:38.740433 (+ 10074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.741488 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.741498 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.741504 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.741729 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.742305 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.742313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.742319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.742792 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:38.743145 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:38.743865 (+ 720us) server_negotiation.cc:299] Negotiation successful
0926 02:58:38.744319 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:58:38.749574 10834 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:58:38.753048 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.707961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.19.185.253:33491)
0926 02:58:38.708155 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:38.708160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:38.717974 (+ 9814us) server_negotiation.cc:407] Connection header received
0926 02:58:38.719229 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:38.719236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:38.719341 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:38.719413 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:38.721889 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.723051 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.725153 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.725917 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.726026 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.730017 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:38.730027 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:38.730035 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:38.730092 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:38.739610 (+ 9518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.749117 (+ 9507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.749127 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.749133 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.749341 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.750819 (+ 1478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.750825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.750828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.751201 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:38.751460 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:38.752402 (+ 942us) server_negotiation.cc:299] Negotiation successful
0926 02:58:38.752674 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 02:58:38.759620 10746 tablet.cc:2369] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:58:38.754034 10576 master_runner.cc:186] Error getting master registration for 127.19.185.253:33491: OK, instance_id { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" instance_seqno: 1790391518005217 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:58:38.767663 10576 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:58:38.768318 10767 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a [term 1 FOLLOWER]: Refusing update from remote peer fb7e7ee48b3747998866814cce9be049: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:58:38.769433 10819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:58:38.771831 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.718508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.19.185.252:43353)
0926 02:58:38.718737 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:38.718744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:38.732060 (+ 13316us) server_negotiation.cc:407] Connection header received
0926 02:58:38.732804 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:38.732810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:38.732914 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:38.732998 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:38.736711 (+ 3713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.737790 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.746653 (+ 8863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.747549 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.747650 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.764035 (+ 16385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:38.764060 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:38.764065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:38.764111 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:38.768341 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.769187 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.769195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.769198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.769377 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.769962 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.769968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.769971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.770282 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:38.770558 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:38.771134 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 02:58:38.771418 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 02:58:38 dist-test-slave-ktng krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391517, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.185.253@KRBTEST.COM for kudu/127.19.185.252@KRBTEST.COM
I20260926 02:58:38.786489 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.737554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.19.185.252:43353)
0926 02:58:38.758031 (+ 20477us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:38.758037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:38.758061 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:58:38.758121 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:38.758129 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:38.758209 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:38.760889 (+ 2680us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:38.761179 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.762519 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.770881 (+ 8362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.771640 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.773856 (+ 2216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.775255 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:58:38.775264 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:58:38.775271 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:58:38.775336 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:58:38.782770 (+ 7434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.783733 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.783741 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.783744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.783917 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:58:38.784421 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:58:38.784426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:58:38.784430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:58:38.784842 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:58:38.785584 (+ 742us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:38.785795 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 02:58:38.786049 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":20307,"thread_start_us":20742,"threads_started":1}
I20260926 02:58:38.789502 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:38.736769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.185.252:43353 (local address 127.0.0.1:57954)
0926 02:58:38.737221 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 02:58:38.737239 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:58:38.737338 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:58:38.759949 (+ 22611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:58:38.759955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:58:38.760488 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:58:38.760710 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.760726 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.769234 (+ 8508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.769241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.770667 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:58:38.770685 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:58:38.771834 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:58:38.771842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:58:38.772087 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:38.772695 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:58:38.772719 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:58:38.775007 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:58:38.782972 (+ 7965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.782978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.782983 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.783558 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.784057 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:58:38.784063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:58:38.784067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:58:38.784247 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:58:38.784992 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:58:38.784998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:58:38.785206 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 02:58:38.785393 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 02:58:38.789089 (+ 3696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331}
I20260926 02:58:38.797477 10717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3: Bootstrap starting.
I20260926 02:58:38.809788 10717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3: Neither blocks nor log segments found. Creating new log.
I20260926 02:58:38.811139 10717 log.cc:824] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:58:38.822197 10717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3: No bootstrap required, opened a new log
I20260926 02:58:38.824856 10798 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb7e7ee48b3747998866814cce9be049. Latest consensus state: current_term: 1 leader_uuid: "fb7e7ee48b3747998866814cce9be049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } } }
I20260926 02:58:38.825258 10798 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:58:38.829533 10836 mvcc.cc:205] Tried to move back new op lower bound from 7333443660852068352 to 7333443660522070016. Current Snapshot: MvccSnapshot[applied={T|T < 7333443660852068352}]
I20260926 02:58:38.831964 10837 mvcc.cc:205] Tried to move back new op lower bound from 7333443660852068352 to 7333443660522070016. Current Snapshot: MvccSnapshot[applied={T|T < 7333443660852068352}]
I20260926 02:58:38.845620 10798 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb7e7ee48b3747998866814cce9be049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } } }
I20260926 02:58:38.845894 10798 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:58:38.845917 10812 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb7e7ee48b3747998866814cce9be049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } } }
I20260926 02:58:38.846225 10812 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:38.855470 10826 catalog_manager.cc:1347] Generated new cluster ID: 9dfd979dd1f54578a855cc8413fee8c7
I20260926 02:58:38.855585 10826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:58:38.860580 10717 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } }
I20260926 02:58:38.861081 10717 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6efcfdd227e43f4874881244167d8e3, State: Initialized, Role: FOLLOWER
I20260926 02:58:38.864403 10717 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } }
I20260926 02:58:38.882819 10842 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } } }
I20260926 02:58:38.885742 10842 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:58:38.885553 10717 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:58:38.886005 10691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:58:38.889946 10691 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3 [term 1 FOLLOWER]: Refusing update from remote peer fb7e7ee48b3747998866814cce9be049: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:58:38.890561 10692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb7e7ee48b3747998866814cce9be049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a6efcfdd227e43f4874881244167d8e3"
I20260926 02:58:38.892210 10819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:58:38.894112 10692 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb7e7ee48b3747998866814cce9be049 in term 1.
I20260926 02:58:38.901023 10826 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:58:38.902980 10826 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:58:38.931401 10826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049: Generated new TSK 0
I20260926 02:58:38.933199 10826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:58:38.968591 10842 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb7e7ee48b3747998866814cce9be049. Latest consensus state: current_term: 1 leader_uuid: "fb7e7ee48b3747998866814cce9be049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } } }
I20260926 02:58:38.968935 10842 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:58:38.969825 10819 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb7e7ee48b3747998866814cce9be049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } } }
I20260926 02:58:38.970101 10819 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7e7ee48b3747998866814cce9be049 [sys.catalog]: This master's current role is: LEADER
I20260926 02:58:38.975530 10847 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb7e7ee48b3747998866814cce9be049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7e7ee48b3747998866814cce9be049" member_type: VOTER last_known_addr { host: "127.19.185.254" port: 38379 } } peers { permanent_uuid: "a6efcfdd227e43f4874881244167d8e3" member_type: VOTER last_known_addr { host: "127.19.185.253" port: 33491 } } peers { permanent_uuid: "e33462cca1804ea7b89c43e2ec41427a" member_type: VOTER last_known_addr { host: "127.19.185.252" port: 43353 } } }
I20260926 02:58:38.976163 10847 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:58:39.058477 10860 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260926 02:58:39.087090 20199 mini_kdc.cc:348] Time spent kdestroy: real 0.142s user 0.000s sys 0.030s
W20260926 02:58:39.141145 10720 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:39.124611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.19.185.254:38379)
0926 02:58:39.124842 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:39.124852 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:39.135791 (+ 10939us) server_negotiation.cc:407] Connection header received
0926 02:58:39.137455 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:39.137465 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:39.137573 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:39.137683 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:39.140795 (+ 3112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49186: BlockingRecv error: recv got EOF from 127.0.0.1:49186 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 02:58:39.145100 10722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:39.136438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44666 (local address 127.19.185.253:33491)
0926 02:58:39.136671 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:39.136680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:39.142659 (+ 5979us) server_negotiation.cc:407] Connection header received
0926 02:58:39.143465 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:39.143474 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:39.143595 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:39.143689 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:39.144785 (+ 1096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44666: BlockingRecv error: recv got EOF from 127.0.0.1:44666 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 02:58:39.151031 10835 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:39.148121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57958 (local address 127.19.185.252:43353)
0926 02:58:39.148362 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:39.148370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:39.148959 (+ 589us) server_negotiation.cc:407] Connection header received
0926 02:58:39.149231 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:39.149240 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:39.149335 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:39.149427 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:39.150686 (+ 1259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57958: BlockingRecv error: recv got EOF from 127.0.0.1:57958 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 02:58:39.227073 10722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:39.213610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.19.185.253:33491)
0926 02:58:39.213820 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:39.213829 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:39.224608 (+ 10779us) server_negotiation.cc:407] Connection header received
0926 02:58:39.225218 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:39.225228 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:39.225381 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:39.225478 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:39.226700 (+ 1222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44678: BlockingRecv error: recv got EOF from 127.0.0.1:44678 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 02:58:39.278474 10722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:39.264798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44690 (local address 127.19.185.253:33491)
0926 02:58:39.265005 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:39.265012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:39.275963 (+ 10951us) server_negotiation.cc:407] Connection header received
0926 02:58:39.276675 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:39.276683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:39.276810 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:39.276901 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:39.278202 (+ 1301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44690: BlockingRecv error: recv got EOF from 127.0.0.1:44690 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 02:58:39.327170 10722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:39.316200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.19.185.253:33491)
0926 02:58:39.316411 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:39.316420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:39.325320 (+ 8900us) server_negotiation.cc:407] Connection header received
0926 02:58:39.325558 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:39.325564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:39.325706 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:39.325854 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:39.326872 (+ 1018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44694: BlockingRecv error: recv got EOF from 127.0.0.1:44694 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 02:58:39.338757 10722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:39.336538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.19.185.253:33491)
0926 02:58:39.336814 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:39.336821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:39.336946 (+ 125us) server_negotiation.cc:407] Connection header received
0926 02:58:39.337302 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:39.337312 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:39.337421 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:39.337551 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:39.338486 (+ 935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44706: BlockingRecv error: recv got EOF from 127.0.0.1:44706 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
W20260926 02:58:39.353976 10722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:39.351579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.19.185.253:33491)
0926 02:58:39.351806 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:39.351814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:39.352284 (+ 470us) server_negotiation.cc:407] Connection header received
0926 02:58:39.352508 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:39.352515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:39.352618 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:39.352740 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:39.353688 (+ 948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44722: BlockingRecv error: recv got EOF from 127.0.0.1:44722 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 02:58:39.377767 10722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:39.374837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44730 (local address 127.19.185.253:33491)
0926 02:58:39.375072 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:39.375080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:39.375689 (+ 609us) server_negotiation.cc:407] Connection header received
0926 02:58:39.376047 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:39.376056 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:39.376173 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:39.376259 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:39.377449 (+ 1190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44730: BlockingRecv error: recv got EOF from 127.0.0.1:44730 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 02:58:39.423276 10722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:39.415042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.19.185.253:33491)
0926 02:58:39.415259 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:39.415265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:39.421487 (+ 6222us) server_negotiation.cc:407] Connection header received
0926 02:58:39.421719 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:39.421729 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:39.421859 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:39.422011 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:39.422953 (+ 942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44742: BlockingRecv error: recv got EOF from 127.0.0.1:44742 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260926 02:58:39.493358 10722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:39.491145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.19.185.253:33491)
0926 02:58:39.491453 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:39.491464 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:39.492097 (+ 633us) server_negotiation.cc:407] Connection header received
0926 02:58:39.492270 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:39.492274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:39.492352 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:39.492456 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:39.493196 (+ 740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44748: BlockingRecv error: recv got EOF from 127.0.0.1:44748 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:58:39.623782 10809 catalog_manager.cc:1259] Loaded cluster ID: 9dfd979dd1f54578a855cc8413fee8c7
I20260926 02:58:39.623884 10809 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a: loading cluster ID for follower catalog manager: success
I20260926 02:58:39.626752 10809 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a: acquiring CA information for follower catalog manager: success
W20260926 02:58:39.627864 10722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:39.625630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.19.185.253:33491)
0926 02:58:39.625959 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:39.625968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:39.626235 (+ 267us) server_negotiation.cc:407] Connection header received
0926 02:58:39.626484 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:39.626488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:39.626566 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:39.626678 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:39.627576 (+ 898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44762: BlockingRecv error: recv got EOF from 127.0.0.1:44762 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:58:39.629007 10809 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e33462cca1804ea7b89c43e2ec41427a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 02:58:39.890298 10722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:58:39.888481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.19.185.253:33491)
0926 02:58:39.888661 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:39.888666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:39.888886 (+ 220us) server_negotiation.cc:407] Connection header received
0926 02:58:39.889040 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:39.889044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:39.889121 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:39.889246 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:58:39.890027 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44770: BlockingRecv error: recv got EOF from 127.0.0.1:44770 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:58:40.063763 10860 catalog_manager.cc:1259] Loaded cluster ID: 9dfd979dd1f54578a855cc8413fee8c7
I20260926 02:58:40.063917 10860 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3: loading cluster ID for follower catalog manager: success
I20260926 02:58:40.067888 10860 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3: acquiring CA information for follower catalog manager: success
I20260926 02:58:40.070479 10860 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a6efcfdd227e43f4874881244167d8e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:58:40.419420 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:40.412528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.19.185.253:33491)
0926 02:58:40.413494 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:40.413501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:40.413521 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:58:40.413603 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:40.413607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:40.413747 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:40.413905 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:40.414456 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:40.415374 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:40.416896 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:40.417395 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:40.417545 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:40.418103 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:40.418663 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:40.418855 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:40.418966 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 02:58:40.419053 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":640,"threads_started":1}
I20260926 02:58:40.433465 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:40.426135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.19.185.252:43353)
0926 02:58:40.426681 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:40.426686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:40.426704 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:58:40.426894 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:40.426899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:40.427069 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:40.427232 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:40.427889 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:40.429117 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:40.430567 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:40.431253 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:40.431414 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:40.432055 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:40.432770 (+ 715us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:40.432927 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:40.433042 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 02:58:40.433134 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":316,"threads_started":1}
I20260926 02:58:40.449390 10935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:40.440719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.19.185.254:38379)
0926 02:58:40.442211 (+ 1492us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:40.442216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:40.442233 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:58:40.442341 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:40.442345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:40.442507 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:40.442744 (+ 237us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:40.443537 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:40.444703 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:40.446623 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:40.447364 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:40.447509 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:40.448085 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:40.448622 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:40.448727 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:40.448888 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 02:58:40.448995 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1393,"thread_start_us":1283,"threads_started":1}
I20260926 02:58:40.449676 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:40.442903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.19.185.252:43353)
0926 02:58:40.443124 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:40.443131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:40.443867 (+ 736us) server_negotiation.cc:407] Connection header received
0926 02:58:40.444097 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:40.444103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:40.444197 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:40.444288 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:40.444851 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:40.445850 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:40.447603 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:40.448142 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:40.448214 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:40.448757 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:40.448964 (+ 207us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:40.449056 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:40.449289 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 02:58:40.449379 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 02:58:40.449843 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:58:40.441765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.19.185.253:33491)
0926 02:58:40.441986 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 02:58:40.441995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:58:40.442672 (+ 677us) server_negotiation.cc:407] Connection header received
0926 02:58:40.443272 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:58:40.443278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:58:40.443410 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:58:40.443509 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:58:40.444239 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:40.445398 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:40.447782 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:58:40.448419 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:58:40.448560 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:58:40.449054 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:58:40.449204 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:58:40.449293 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 02:58:40.449453 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 02:58:40.449536 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 02:58:40.454461 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 10576
I20260926 02:58:40.456043 10638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:58:40.576081 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 10576
I20260926 02:58:40.598069 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 10650
I20260926 02:58:40.599412 10712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:58:40.706777 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 10650
I20260926 02:58:40.727636 20199 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu pid 10725
I20260926 02:58:40.728786 10787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:58:40.836506 20199 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAXkdvX/build/asan/bin/kudu with pid 10725
2026-09-26T02:58:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5256 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9746 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (369257 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST